Databricks Data Engineer: Lakehouse & Spark Architect

CloudBigD

Atlanta (GA)

On-site

USD 86,000 - 123,000

Full time

1 hour ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

CloudBigD is seeking a Databricks Data Engineer to design and manage enterprise-scale data processing solutions on Databricks, Spark, PySpark, and Delta Lake in Atlanta, GA. You will build batch and streaming data pipelines, implement lakehouse architecture, and ensure data governance and quality across platforms.

Ideal candidates have hands-on Databricks experience, strong Python/PySpark skills, and familiarity with AWS/Azure/GCP, SQL, ETL/ELT, and CI/CD.

Qualifications

  • Proven experience with Databricks in enterprise-level data platforms.
  • Strong PySpark coding skills and Python fluency.
  • Hands-on knowledge of Spark SQL and Delta Lake.
  • Experience implementing lakehouse architectures and data governance.
  • Familiarity with Unity Catalog and Delta Live Tables workflows.

Responsibilities

  • Design scalable data engineering solutions using Databricks.
  • Build batch and streaming pipelines with Spark.
  • Develop data transformations using PySpark and Spark SQL.
  • Implement lakehouse architecture using Delta Lake.
  • Manage data access, governance, and permissions via Unity Catalog.
  • Build automated workflows using Databricks Workflows.
  • Implement incremental processing and CDC pipelines.
  • Optimize Spark jobs for performance.
  • Integrate Databricks with cloud storage and databases.
  • Develop data validation and quality monitoring processes.
  • Collaborate with data scientists and analytics teams.
  • Troubleshoot production issues to ensure pipeline reliability.

Skills

Databricks
Python/PySpark
Spark SQL
Delta Lake
Unity Catalog
Delta Live Tables
ETL/ELT pipelines
Cloud platforms
SQL and data warehousing
Git CI/CD

Tools

Terraform
Databricks Workflows
Apache Kafka
MLflow

Job description

CloudBigD is seeking a Databricks Data Engineer to design and manage enterprise-scale data processing solutions on Databricks, Spark, PySpark, and Delta Lake in Atlanta, GA. You will build batch and streaming data pipelines, implement lakehouse architecture, and ensure data governance and quality across platforms.

Ideal candidates have hands-on Databricks experience, strong Python/PySpark skills, and familiarity with AWS/Azure/GCP, SQL, ETL/ELT, and CI/CD.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Databricks Data Engineer: Lakehouse & Spark Expert
Databricks Data Engineer: Lakehouse & Spark Expert

CloudBigD • Louisville (KY)

On-site
USD 120,000 - 180,000
Databricks Data Engineer: Lakehouse & Streaming Specialist
Databricks Data Engineer: Lakehouse & Streaming Specialist

CloudBigD • Houston (TX)

On-site
USD 120,000 - 180,000
Lead Data Architect & Engineer - Databricks Lakehouse
Lead Data Architect & Engineer - Databricks Lakehouse

CloudBigD • Atlanta (GA)

On-site
USD 89,000 - 126,000
Databricks Data Engineer — Lakehouse & Governance Expert
Databricks Data Engineer — Lakehouse & Governance Expert

UNAVAILABLE • McLean (VA)

On-site
USD 120,000 - 170,000
Databricks Data Engineer - Spark, ETL & Lakehouse
Databricks Data Engineer - Spark, ETL & Lakehouse

CloudBigD • Quincy (MA)

On-site
USD 77,000 - 110,000
Senior Data Engineer - Databricks Lakehouse Architect
Senior Data Engineer - Databricks Lakehouse Architect

UNAVAILABLE • McLean (VA)

On-site
USD 120,000 - 160,000
Senior Data Engineer: Databricks Lakehouse Leader
Senior Data Engineer: Databricks Lakehouse Leader

Tata Consultancy Services Limited • Irvine (CA)

On-site
USD 120,000 - 180,000
Databricks Data Engineer - Lakehouse & Delta Lead
Databricks Data Engineer - Lakehouse & Delta Lead

Jobgether • United States

Remote
USD 110,000 - 160,000
Healthcare coverage
Meal vouchers
Life insurance
+4
Senior Data Engineer, Databricks Lakehouse (Onsite Atlanta)
Senior Data Engineer, Databricks Lakehouse (Onsite Atlanta)

Capgemini • Atlanta (GA)

On-site
USD 89,000 - 166,000
Vacation: 12-25 days
Company paid holidays
Personal days
+5
Senior Databricks Lakehouse Architect (PySpark/ETL)
Senior Databricks Lakehouse Architect (PySpark/ETL)

VService IT Solutions • United States

Remote
USD 140,000 - 190,000