Databricks Data Engineer - Azure, CDC Pipelines, PySpark

CloudBigD

Phoenix (AZ)

Presencial

USD 81.000 - 115.000

Jornada completa

hace 4 horas
Sé de los primeros/as/es en solicitar esta vacante

Una candidatura completa en un minuto

Currículum y carta de presentación personalizados, listos para enviar.

Supera los filtros ATS

Descripción de la vacante

CloudBigD in Phoenix, AZ is seeking a Databricks Data Engineer to design, build and optimize scalable data solutions. The role focuses on end-to-end ETL/ELT pipelines, CDC-enabled processing, and real-time data streams using PySpark, Kafka, and Delta Lake in an Azure environment.

You will work across new and existing pipelines, integrate data from diverse sources, implement data quality rules, and participate in Agile ceremonies to deliver high-quality solutions within a 9+ month engagement.

Formación

  • Extensive hands-on Databricks experience.
  • End-to-end data pipelines with CDC capabilities.
  • Strong PySpark and SQL; Python is a plus.
  • Experience in Microsoft Azure cloud environments.

Responsabilidades

  • Design, build, and optimize end-to-end ETL/ELT pipelines using Databricks and Apache Spark (PySpark).
  • Develop and maintain scalable data solutions on Microsoft Azure, leveraging Databricks, Delta Lake, and Kafka streaming.
  • Build robust pipelines with Change Data Capture (CDC) to support incremental data processing and real-time/near real-time data needs.
  • Work across both new pipeline development and enhancement of existing pipelines, quickly understanding current codebases to deliver improvements.
  • Integrate and process data from diverse sources including: Flat files, XML, JSON, SQL Server and JDBC systems, Kafka topics, APIs, MQ, Unity Catalog volumes and other enterprise platforms.
  • Implement data quality frameworks, including writing technical data quality rules and quarantining error records.
  • Apply canonical data modeling techniques, including SCD Type 1 and Type 2 implementations.
  • Ensure high levels of data reliability, performance, scalability, and monitoring across all workflows.
  • Automate workflows using Databricks Workflows and CI/CD pipelines (GitHub), focusing on reuse, efficiency, and standardization.
  • Collaborate with developers, architects, and product owners, and engage with stakeholders, source teams, and downstream consumers as needed.
  • Participate actively in Agile/Scrum ceremonies and deliver high-quality solutions within defined timelines.

Conocimientos

Databricks
PySpark
SQL
Python
CDC
Kafka
Azure

Herramientas

Delta Lake
Unity Catalog
Kafka Streaming

Descripción del empleo

CloudBigD in Phoenix, AZ is seeking a Databricks Data Engineer to design, build and optimize scalable data solutions. The role focuses on end-to-end ETL/ELT pipelines, CDC-enabled processing, and real-time data streams using PySpark, Kafka, and Delta Lake in an Azure environment.

You will work across new and existing pipelines, integrate data from diverse sources, implement data quality rules, and participate in Agile ceremonies to deliver high-quality solutions within a 9+ month engagement.

Consigue la evaluación confidencial y gratuita de tu currículum.

o arrastra y suelta tu archivo aquí

Similar jobs

Puestos de trabajo similares que vale la pena comparar

Databricks Data Engineer - End-to-End CDC Pipelines (Azure)
Databricks Data Engineer - End-to-End CDC Pipelines (Azure)

CRG • EE. UU.

Presencial
USD 90.000 - 94.000
Databricks Data Engineer: Azure CDC Pipelines
Databricks Data Engineer: Azure CDC Pipelines

Valiant Technologies LLC • Phoenix (AZ)

Presencial
USD 81.000 - 115.000
Azure Databricks Data Engineer - CDC & Spark Expert
Azure Databricks Data Engineer - CDC & Spark Expert

Quantum Technologies. LLC • Phoenix (AZ)

Presencial
USD 81.000 - 115.000
Databricks Data Engineer | PySpark, Delta Lake, Azure
Databricks Data Engineer | PySpark, Delta Lake, Azure

BuzzClan LLC • Beaverton (OR)

Presencial
USD 120.000 - 170.000
Azure Data Engineer
Azure Data Engineer

Signature IT World • Pennsylvania

Híbrido
USD 120.000 - 150.000
Azure Data Engineer
Azure Data Engineer

Signature IT World • New Jersey

Híbrido
USD 110.000 - 150.000
Databricks Data Engineer
Databricks Data Engineer

Valiant Technologies LLC • Phoenix (AZ)

Presencial
USD 81.000 - 115.000
Data Engineer – Databricks & Azure Data Factory
Data Engineer – Databricks & Azure Data Factory

CloudBigD • Plano (TX)

Presencial
USD 98.000 - 139.000
Databricks Data Engineer
Databricks Data Engineer

Quantum Technologies. LLC • Phoenix (AZ)

Presencial
USD 81.000 - 115.000
Databricks Data Engineer
Databricks Data Engineer

CloudBigD • Phoenix (AZ)

Presencial
USD 81.000 - 115.000