Senior Azure Databricks Data Engineer

EXL; Detroit (MI); On-site; USD 120,000 - 190,000; Full time

Job summary

EXL is seeking a Senior Azure Databricks Data Engineer to design, build, and operate enterprise-scale data pipelines and curated data products. You will own data flows from ingestion to consumption, delivering reliable, production-ready solutions using Azure Databricks, PySpark, SQL, Python, and Delta Lake.

You will drive the Medallion architecture (Bronze, Silver, Gold), implement efficient batch and streaming patterns, ensure data quality and governance, and supervise production workloads with

Qualifications

• 6+ years of experience in Data Engineering, with Azure Databricks
• Advanced PySpark / Apache Spark in production
• Advanced SQL skills with complex transformations and performance tuning
• Strong experience with Delta Lake / Lakehouse architectures
• Hands-on implementation of Bronze, Silver, Gold Medallion Architecture
• Experience building production-grade ETL/ELT pipelines
• Experience ingesting data from multiple enterprise sources
• Knowledge of batch, incremental, CDC, watermarking and schema evolution
• Experience with Databricks Unity Catalog, lineage and governance
• Ability to troubleshoot Spark jobs and production pipelines
• Experience with Git-based development and CI/CD practices

Responsibilities

• Design and develop scalable ETL/ELT pipelines using Azure Databricks, PySpark, Python, and SQL.
• Build reusable ingestion and transformation frameworks.
• Develop and maintain Delta Lake tables across Bronze, Silver, and Gold layers.
• Implement batch, incremental, CDC, and watermark-based processing.
• Build production-grade pipelines with logging, monitoring, error handling, and recovery.

Skills

Data engineering

Azure Databricks

PySpark

SQL

Delta Lake

Git

CI/CD

Spark

Tools

Azure Databricks

PySpark

SQL

Delta Lake

Databricks Unity Catalog

Git

CI/CD

Job description

EXL is looking for a Senior Azure Databricks Data Engineer to design, build, and operate enterprise-scale data pipelines and curated data products.

In this role, you will work across the full data lifecycle, from source ingestion and Bronze landing through Silver transformations, data quality, and Gold data-product enablement. You will be responsible for building reliable, scalable, and production-ready solutions using Azure Databricks, PySpark, SQL, Python, and Delta Lake.

We are looking for an experienced engineer who can take ownership of data domains end to end, independently troubleshoot production workloads, and translate business rules and source-to-target requirements into robust data pipelines.

Key Responsibilities

• Design and develop scalable ETL/ELT pipelines using Azure Databricks, PySpark, Python, and SQL.
• Build reusable and configuration-driven ingestion and transformation frameworks.
• Develop and maintain Delta Lake tables across Bronze, Silver, and Gold layers.
• Implement efficient batch, incremental, CDC, and watermark-based processing patterns.
• Build production-grade pipelines with appropriate logging, monitoring, error handling, and recovery mechanisms.

Data Ingestion & Integration

• Integrate data from heterogeneous enterprise sources including Oracle, SAP, Salesforce, relational databases, REST APIs, SFTP, CSV, Excel, and legacy platforms.
• Design incremental loads, change data capture, schema evolution, restart, and reprocessing strategies.
• Capture ingestion metadata, source identifiers, processing status, and audit information to ensure end-to-end traceability.

Medallion Architecture

• Bronze: Land raw source data while preserving history, lineage, traceability, and audit controls.
• Silver: Cleanse, standardize, transform, and deduplicate data while applying business rules and integrity validations.
• Gold: Create consumption-ready datasets and data products supporting KPIs, analytics, reporting, and downstream applications.

Data Quality & Reconciliation

• Build automated controls for completeness, validity, uniqueness, referential integrity, and business-rule validation.
• Implement exception handling and quarantine mechanisms for problematic records.
• Reconcile pipeline outputs against source systems and downstream reporting metrics.
• Perform root-cause analysis and resolve data discrepancies.

Governance & Documentation

• Work with Databricks Unity Catalog to support governed tables, views, access controls, and lineage.
• Maintain data dictionaries, source-to-target mappings, transformation specifications, and operational documentation.
• Promote reusable engineering standards and data governance best practices.

Production Engineering & Performance

• Schedule, monitor, troubleshoot, and support production Databricks workloads.
• Optimize Spark jobs, joins, shuffles, partitioning, caching, and Delta Lake file layouts.
• Design idempotent and restartable pipelines with effective logging, monitoring, and alerting.
• Independently diagnose Spark, data, and pipeline failures and perform root-cause analysis.

Required Qualifications

• 6+ years of experience in Data Engineering, including substantial recent hands‑on experience with Azure Databricks.
• Advanced hands‑on experience with PySpark / Apache Spark in production environments.
• Advanced SQL skills, including complex transformations and performance tuning.
• Strong experience designing and implementing Delta Lake / Lakehouse architectures.
• Hands‑on implementation of Bronze, Silver, and Gold Medallion Architecture.
• Experience building production‑grade ETL/ELT pipelines beyond notebook‑based proofs of concept.
• Experience ingesting data from multiple enterprise source systems.
• Strong understanding of batch, incremental, CDC, watermarking, schema evolution, and reprocessing patterns.
• Experience implementing automated data‑quality, reconciliation, and exception‑handling controls.
• Experience with Databricks Unity Catalog, lineage, and governed data assets.
• Ability to independently troubleshoot Spark jobs and production pipeline failures.
• Experience with Git‑based development and CI/CD practices.

Nice to Have

• Experience with Azure Data Factory (ADF), Databricks Auto Loader, or Lakeflow / declarative pipelines.
• Experience migrating legacy SQL, Microsoft Access, Excel, or VBA processes into scalable cloud data pipelines.
• Experience integrating data from SAP, Oracle, or Salesforce.
• Familiarity with Power BI or other downstream analytics and reporting platforms.
• Databricks or Microsoft Azure certifications.
• Previous experience within manufacturing or automotive environments.

Seniority & Ownership Expectations

As a Senior Data Engineer, you will be expected to:

• Own a data source or business domain end to end, from initial assessment through Bronze, Silver, and Gold enablement.
• Translate source-to-target mappings and business requirements into production‑ready pipelines.
• Make technical decisions around ingestion, transformation, data quality, performance, and recovery patterns.
• Review code from junior engineers and promote reusable engineering standards.
• Collaborate directly with Data Architects, Business Analysts, QA Engineers, and Data Owners.
• Estimate engineering effort and proactively identify source‑data, architecture, or delivery risks.
• Own technical documentation and support effective knowledge transfer and handover.

What We’re Looking For

We are looking for someone who combines strong hands‑on Databricks engineering skills with senior‑level ownership. The ideal candidate is comfortable taking a data pipeline from initial source assessment to production, solving complex data‑quality or performance issues independently, and building solutions that are scalable, maintainable, and ready for enterprise use.