Databricks Data Engineer: Azure CDC Pipelines

Valiant Technologies LLC

Phoenix (AZ)

On-site

USD 81,000 - 115,000

Full time

3 hours ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Valiant Technologies LLC in Phoenix, AZ is seeking a highly skilled Databricks Data Engineer to design, build, and optimize scalable data pipelines for the Retail Spine initiative. You will work with Databricks, PySpark, Spark, and Azure to deliver end-to-end ETL/ELT solutions and ensure data quality.

The role emphasizes CDC, streaming, and real-time data, integrating sources like XML, JSON, SQL Server, Kafka, and APIs, while supporting CI/CD workflows and Agile practices.

Qualifications

  • Extensive hands-on experience with Databricks.
  • End-to-end data pipelines with CDC capabilities.
  • Proficient in PySpark and SQL; Python is a plus.
  • Azure cloud experience.

Responsibilities

  • Design, build, and optimize end-to-end ETL/ELT pipelines with Databricks and PySpark.
  • Develop data solutions on Azure using Databricks, Delta Lake, and Kafka.
  • Implement CDC to enable real-time/incremental processing.
  • Work on new pipelines and enhance existing ones; review codebases for improvements.
  • Integrate data from flat files, XML, JSON, SQL Server, JDBC, Kafka, APIs, MQ, and more.
  • Ensure data quality with rules and quarantining error records.
  • Apply SCD Type 1 and Type 2 canonical data modeling.
  • Automate workflows with Databricks Workflows and GitHub CI/CD.
  • Collaborate with engineers, architects, product owners and stakeholders.
  • Participate in Agile/Scrum ceremonies.

Skills

Databricks
PySpark
SQL
Python
CDC
Kafka
Azure

Tools

Delta Lake
Unity Catalog
Apache Spark
Kafka

Job description

Valiant Technologies LLC in Phoenix, AZ is seeking a highly skilled Databricks Data Engineer to design, build, and optimize scalable data pipelines for the Retail Spine initiative. You will work with Databricks, PySpark, Spark, and Azure to deliver end-to-end ETL/ELT solutions and ensure data quality.

The role emphasizes CDC, streaming, and real-time data, integrating sources like XML, JSON, SQL Server, Kafka, and APIs, while supporting CI/CD workflows and Agile practices.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Databricks Data Engineer - Azure, CDC Pipelines, PySpark
Databricks Data Engineer - Azure, CDC Pipelines, PySpark

CloudBigD • Phoenix (AZ)

On-site
USD 81,000 - 115,000
Databricks Data Engineer - End-to-End CDC Pipelines (Azure)
Databricks Data Engineer - End-to-End CDC Pipelines (Azure)

CRG • United States

On-site
USD 90,000 - 94,000
Senior Azure Data Engineer: Pipelines, Databricks & SQL
Senior Azure Data Engineer: Pipelines, Databricks & SQL

Valiant Technologies LLC • Houston (TX)

On-site
USD 120,000 - 170,000
Azure Data Engineer: Pipelines, Data Lakes & CDC
Azure Data Engineer: Pipelines, Data Lakes & CDC

Valiant Technologies LLC • Grand Rapids (MI)

On-site
USD 110,000 - 150,000
Data Engineer: Databricks & ADF Pipelines
Data Engineer: Databricks & ADF Pipelines

Valiant Technologies LLC • Atlanta (GA)

On-site
USD 90,000 - 130,000
Azure Databricks Engineer: Data Pipelines & Governance
Azure Databricks Engineer: Data Pipelines & Governance

Valiant Technologies LLC • Cincinnati (OH)

On-site
USD 90,000 - 110,000
Azure Databricks Data Engineer - CDC & Spark Expert
Azure Databricks Data Engineer - CDC & Spark Expert

Quantum Technologies. LLC • Phoenix (AZ)

On-site
USD 81,000 - 115,000
Databricks Data Engineer
Databricks Data Engineer

CRG • United States

On-site
USD 90,000 - 94,000
Azure Data Engineer - Databricks, PySpark & Healthcare Data
Azure Data Engineer - Databricks, PySpark & Healthcare Data

Valiant Technologies LLC • Cupertino (CA)

On-site
USD 120,000 - 170,000
Senior Azure Databricks Data Engineer
Senior Azure Databricks Data Engineer

Valiant Technologies LLC • Houston (TX)

On-site
USD 110,000 - 160,000