Senior Azure Databricks Data Engineer: Delta Lake Pipelines

EXL; Detroit (MI); On-site; USD 120,000 - 190,000; Full time

Job summary

EXL is seeking a Senior Azure Databricks Data Engineer to design, build, and operate enterprise-scale data pipelines and curated data products. You will own data flows from ingestion to consumption, delivering reliable, production-ready solutions using Azure Databricks, PySpark, SQL, Python, and Delta Lake.

You will drive the Medallion architecture (Bronze, Silver, Gold), implement efficient batch and streaming patterns, ensure data quality and governance, and supervise production workloads with

Qualifications

• 6+ years of experience in Data Engineering, with Azure Databricks
• Advanced PySpark / Apache Spark in production
• Advanced SQL skills with complex transformations and performance tuning
• Strong experience with Delta Lake / Lakehouse architectures
• Hands-on implementation of Bronze, Silver, Gold Medallion Architecture
• Experience building production-grade ETL/ELT pipelines
• Experience ingesting data from multiple enterprise sources
• Knowledge of batch, incremental, CDC, watermarking and schema evolution
• Experience with Databricks Unity Catalog, lineage and governance
• Ability to troubleshoot Spark jobs and production pipelines
• Experience with Git-based development and CI/CD practices

Responsibilities

• Design and develop scalable ETL/ELT pipelines using Azure Databricks, PySpark, Python, and SQL.
• Build reusable ingestion and transformation frameworks.
• Develop and maintain Delta Lake tables across Bronze, Silver, and Gold layers.
• Implement batch, incremental, CDC, and watermark-based processing.
• Build production-grade pipelines with logging, monitoring, error handling, and recovery.

Skills

Data engineering

Azure Databricks

PySpark

SQL

Delta Lake

Git

CI/CD

Spark

Tools

Azure Databricks

PySpark

SQL

Delta Lake

Databricks Unity Catalog

Git

CI/CD

Job description

EXL is seeking a Senior Azure Databricks Data Engineer to design, build, and operate enterprise-scale data pipelines and curated data products. You will own data flows from ingestion to consumption, delivering reliable, production-ready solutions using Azure Databricks, PySpark, SQL, Python, and Delta Lake.

You will drive the Medallion architecture (Bronze, Silver, Gold), implement efficient batch and streaming patterns, ensure data quality and governance, and supervise production workloads with