Azure Databricks Data Engineer - CDC & Spark Expert

Quantum Technologies. LLC

Phoenix (AZ)

On-site

USD 81,000 - 115,000

Full time

4 hours ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Quantum Technologies. LLC is seeking a Databricks Data Engineer to design, build, and optimize scalable data pipelines in a fast-paced Azure-based environment.

You will work with Databricks, PySpark, Delta Lake, and Kafka to deliver end-to-end ETL/ELT solutions and CDC-enabled data processing. The role requires strong PySpark and SQL skills, hands-on Databricks experience, and experience with data quality, SCD modeling, and CI/CD workflows in GitHub.

Qualifications

  • Hands-on Databricks experience with data engineering.
  • Experience building CDC-enabled end-to-end pipelines.
  • Strong PySpark and SQL; Python a plus.
  • Azure cloud environment experience is required.
  • Familiarity with ETL/ELT design and data modeling.

Responsibilities

  • Design, build, and optimize end-to-end ETL/ELT pipelines with Databricks and Spark.
  • Develop and maintain data solutions on Azure using Databricks, Delta Lake, and Kafka.
  • Build CDC-based incremental data processing for real-time needs.
  • Work on new pipelines and enhance existing ones; review codebases.
  • Integrate data from flat files, XML, JSON, SQL Server, JDBC, APIs, MQ.
  • Implement data quality rules and quarantine error records.
  • Apply SCD Type 1 and Type 2 modelling techniques.
  • Ensure reliability, performance, and monitoring across workflows.
  • Automate workflows with Databricks Workflows and CI/CD (GitHub).
  • Collaborate with stakeholders and participate in Agile ceremonies.

Skills

CDC capabilities
End-to-end data pipelines
PySpark
SQL
Azure cloud experience
Databricks expertise

Tools

Databricks
Delta Lake
Kafka streaming
Azure
SQL Server
JDBC

Job description

Quantum Technologies. LLC is seeking a Databricks Data Engineer to design, build, and optimize scalable data pipelines in a fast-paced Azure-based environment.

You will work with Databricks, PySpark, Delta Lake, and Kafka to deliver end-to-end ETL/ELT solutions and CDC-enabled data processing. The role requires strong PySpark and SQL skills, hands-on Databricks experience, and experience with data quality, SCD modeling, and CI/CD workflows in GitHub.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Databricks Data Engineer - Azure, CDC Pipelines, PySpark
Databricks Data Engineer - Azure, CDC Pipelines, PySpark

CloudBigD • Phoenix (AZ)

On-site
USD 81,000 - 115,000
Azure Data Engineer – ETL, Databricks & Data Lake Architect
Azure Data Engineer – ETL, Databricks & Data Lake Architect

Quantum Technologies. LLC • Atlanta (GA)

On-site
USD 120,000 - 150,000
Data Engineer - Databricks & Spark on Azure
Data Engineer - Databricks & Spark on Azure

Quantum Technologies LLC • Quincy (MA)

On-site
USD 80,000 - 107,000
Data Engineer — Databricks & ADF Pipelines
Data Engineer — Databricks & ADF Pipelines

Quantum Technologies. LLC • Atlanta (GA)

On-site
USD 90,000 - 150,000
Azure Databricks Data Engineer — Lakehouse & Spark Expert
Azure Databricks Data Engineer — Lakehouse & Spark Expert

Quantum Technologies LLC • Dallas (TX)

On-site
USD 120,000 - 180,000
Databricks Data Engineer - End-to-End CDC Pipelines (Azure)
Databricks Data Engineer - End-to-End CDC Pipelines (Azure)

CRG • United States

On-site
USD 90,000 - 94,000
Data Engineer - Spark, Databricks & Azure ETL Expert
Data Engineer - Spark, Databricks & Azure ETL Expert

Quantum Technologies LLC • Plano (TX)

On-site
USD 98,000 - 139,000
Data Engineer: Spark, Databricks & Lakehouse Architect
Data Engineer: Spark, Databricks & Lakehouse Architect

Quantum Technologies LLC • Houston (TX)

On-site
USD 110,000 - 170,000
Azure Data Engineer — Lead Data Pipelines & Databricks
Azure Data Engineer — Lead Data Pipelines & Databricks

Quantum Technologies LLC • Pennsylvania

On-site
USD 91,000 - 124,000
Azure Databricks Engineer: Governance, PySpark & Pipelines
Azure Databricks Engineer: Governance, PySpark & Pipelines

Quantum Technologies LLC • Cincinnati (OH)

On-site
USD 110,000 - 165,000