Databricks Data Engineer

CloudBigD

Atlanta (GA)

Presencial

USD 86.000 - 123.000

Tempo integral

há 3 horas
Seja uma das primeiras pessoas a se candidatar

Uma candidatura completa em um minuto

Currículo e carta de apresentação personalizados, prontos para serem enviados.

Passe pelos filtros ATS

Resumo da vaga

CloudBigD is seeking a Databricks Data Engineer to design and manage enterprise-scale data processing solutions on Databricks, Spark, PySpark, and Delta Lake in Atlanta, GA. You will build batch and streaming data pipelines, implement lakehouse architecture, and ensure data governance and quality across platforms.

Ideal candidates have hands-on Databricks experience, strong Python/PySpark skills, and familiarity with AWS/Azure/GCP, SQL, ETL/ELT, and CI/CD.

Qualificações

  • Proven experience with Databricks in enterprise-level data platforms.
  • Strong PySpark coding skills and Python fluency.
  • Hands-on knowledge of Spark SQL and Delta Lake.
  • Experience implementing lakehouse architectures and data governance.
  • Familiarity with Unity Catalog and Delta Live Tables workflows.

Responsabilidades

  • Design scalable data engineering solutions using Databricks.
  • Build batch and streaming pipelines with Spark.
  • Develop data transformations using PySpark and Spark SQL.
  • Implement lakehouse architecture using Delta Lake.
  • Manage data access, governance, and permissions via Unity Catalog.
  • Build automated workflows using Databricks Workflows.
  • Implement incremental processing and CDC pipelines.
  • Optimize Spark jobs for performance.
  • Integrate Databricks with cloud storage and databases.
  • Develop data validation and quality monitoring processes.
  • Collaborate with data scientists and analytics teams.
  • Troubleshoot production issues to ensure pipeline reliability.

Conhecimentos

Databricks
Python/PySpark
Spark SQL
Delta Lake
Unity Catalog
Delta Live Tables
ETL/ELT pipelines
Cloud platforms
SQL and data warehousing
Git CI/CD

Ferramentas

Terraform
Databricks Workflows
Apache Kafka
MLflow

Descrição da vaga

US Citizen, Green Card, H-1B, OPT-EAD, CPT, Other

Job Title: Databricks Data Engineer
Location: Atlanta, GA
Duration: 24 Months + Extension
Bill Rate: $76/hour
Job Type: W-2, 1099, C2C
Client: To Be Discussed Later
Work Authorization: US-Citizen, H-1B, OPT-EAD, GC-EAD
Job Description
  • We are looking for a skilled Databricks Data Engineer to develop and manage enterprise-scale data processing solutions using Databricks, Apache Spark, PySpark, and Delta Lake.
  • The ideal candidate should have strong experience in building data pipelines, implementing lakehouse architecture, optimizing Spark workloads, and integrating cloud data platforms. The candidate will work with data architects, analysts, and engineering teams to deliver scalable and reliable data solutions.
Required Skills:
  • Strong hands-on experience with Databricks.
  • Advanced programming skills in Python and PySpark.
  • Strong knowledge of Apache Spark and Spark SQL.
  • Experience with Delta Lake and lakehouse architecture.
  • Knowledge of Unity Catalog and data governance.
  • Experience with Delta Live Tables / Lakeflow.
  • Experience building ETL and ELT pipelines.
  • Knowledge of AWS, Azure, or GCP cloud platforms.
  • Experience with SQL, data modeling, and data warehousing.
  • Familiarity with Git, CI/CD, and workflow orchestration.
Key Responsibilities:
  • Design and develop scalable data engineering solutions using Databricks.
  • Build batch and streaming data pipelines using Apache Spark.
  • Develop data transformations using PySpark and Spark SQL.
  • Implement lakehouse architecture using Delta Lake.
  • Manage data access, governance, and permissions through Unity Catalog.
  • Build and maintain automated workflows using Databricks Workflows.
  • Implement incremental data processing and CDC pipelines.
  • Optimize Spark jobs and improve processing performance.
  • Integrate Databricks with cloud storage and enterprise databases.
  • Develop data validation and quality monitoring processes.
  • Collaborate with data scientists and analytics teams.
  • Troubleshoot production issues and ensure pipeline reliability.
Preferred Qualifications:
  • Experience with Databricks Lakeflow and Lakeflow Connect.
  • Knowledge of Apache Kafka and real-time data processing.
  • Experience with Terraform and cloud infrastructure automation.
  • Knowledge of MLflow and machine learning workflows.
  • Experience with enterprise data migration projects.

We are an equal opportunity employer. All aspects of employment including the decision to hire, promote, discipline, or discharge, will be based on merit, competence, performance, and business needs. We do not discriminate on the basis of race, color, religion, marital status, age, national origin, ancestry, physical or mental disability, medical condition, pregnancy, genetic information, gender, sexual orientation, gender identity or expression, national origin, citizenship/ immigration status, veteran status, or any other status protected under federal, state, or local law.

Obtenha sua revisão de currículo grátis e confidencial.

ou arraste e solte seu arquivo aqui.

Similar jobs

Vagas similares que valem a pena comparar

Databricks Developer
Databricks Developer

Valiant Technologies LLC • Louisville (KY)

Presencial
USD 140.000 - 190.000
Databricks Engineer
Databricks Engineer

Ethereum Technologies LLC • Austin (TX)

Presencial
USD 130.000 - 185.000
Databricks Engineer
Databricks Engineer

Quantum Technologies LLC • Austin (TX)

Presencial
USD 120.000 - 190.000
Databricks Engineer
Databricks Engineer

Valiant Technologies LLC • Austin (TX)

Presencial
USD 130.000 - 190.000
Senior Data Engineer – Databricks & AWS
Senior Data Engineer – Databricks & AWS

CloudBigD • Sunnyvale (CA)

Presencial
USD 76.000 - 108.000
Databricks Developer
Databricks Developer

Robotics Technologies • Louisville (KY)

Presencial
USD 90.000 - 130.000
Senior Data Engineer – Databricks & AWS
Senior Data Engineer – Databricks & AWS

Ethereum Technologies LLC • Sunnyvale (CA)

Presencial
USD 76.000 - 108.000
Senior Data Engineer – Databricks & AWS
Senior Data Engineer – Databricks & AWS

Robotic Process Automation LLC • Sunnyvale (CA)

Presencial
USD 83.000 - 102.000
Senior Data Engineer – Databricks & AWS
Senior Data Engineer – Databricks & AWS

Valiant Technologies LLC • Sunnyvale (CA)

Presencial
USD 83.000 - 103.000
Senior Data Engineer – Databricks & AWS
Senior Data Engineer – Databricks & AWS

Quantum Technologies LLC • Sunnyvale (CA)

Presencial
USD 76.000 - 108.000