Senior Data Engineer - Databricks

EXL; United States; On-site; USD 63,000 - 86,500; Full time

Job summary

EXL Service is seeking a Senior Databricks Data Engineer with 10–12 years of experience to design, develop, and optimize enterprise data platforms and large-scale ETL solutions. The role emphasizes Databricks, Spark, Delta Lake, and Azure Data Platform to deliver analytics-ready datasets for reporting, AI/ML, and analytics initiatives.

The candidate will build scalable data pipelines, real-time architectures, and reusable data integration frameworks, while ensuring data quality, security, and

Qualifications

• 10-12 years of data engineering experience with enterprise-scale ETL and data platforms.
• Deep expertise in Databricks, Spark, Delta Lake, and Azure Data Platform.
• Proficiency with PySpark, Spark SQL, Python, and SQL.
• Experience building batch, real-time, and CDC data pipelines in cloud environments.
• Certifications in Azure Data Engineer or Databricks preferred.

Responsibilities

• Design, develop, and optimize large-scale data pipelines using Databricks, Spark, and Delta Lake.
• Implement cloud-native data solutions and improve platform performance.
• Build scalable ingestion frameworks for diverse data sources and formats.
• Develop data quality, lineage, monitoring, and error handling.
• Develop Databricks Workflows, SQL Jobs, and automation for production workloads.
• Integrate REST APIs and process JSON/XML data for analytics-ready datasets.
• Collaborate across teams to ensure reliable data products for analytics, reporting, and AI/ML.

Skills

Databricks

Spark

Delta Lake

Azure Data Platform

PySpark

Spark SQL

Python

Azure DevOps

Git

Ab Initio

Tools

Azure DevOps

Ab Initio

Job description

EXL Service is seeking an accomplished Senior Databricks Data Engineer with 10–12 years of experience designing, developing, and optimizing enterprise data platforms and large-scale ETL solutions. The ideal candidate brings deep expertise in Databricks, Spark, Delta Lake, Azure Data Platform, and modern data engineering practices.

This role is responsible for building scalable data pipelines, implementing cloud-native data solutions, improving platform performance, and delivering reliable data products for analytics, reporting, and AI/ML initiatives.

Key Responsibilities

• Design, develop, and optimize enterprise-scale data pipelines using Databricks, Spark, Delta Lake, and Azure Data Lake Storage.
• Build batch and incremental ETL/ELT pipelines, real-time streaming architectures, and reusable data integration frameworks.
• Develop scalable ingestion frameworks supporting structured, semi-structured, and API-based data sources.
• Build reusable data engineering frameworks for ingestion, transformation, validation, reconciliation, and publishing.
• Develop Delta Lake solutions using partitioning, optimization, Z-Ordering, Liquid Clustering, and performance tuning techniques.
• Implement CDC, incremental processing, merge strategies, and data synchronization across enterprise platforms.
• Develop Databricks Workflows, notebooks, SQL Jobs, and automation for production workloads.
• Integrate external REST APIs and process JSON/XML data into analytics-ready datasets.
• Implement robust data quality checks, audit frameworks, monitoring, and error handling.
• Optimize Spark jobs for performance, scalability, and cost efficiency.
• Develop solutions using Azure Data Lake Storage Gen2, Databricks, Unity Catalog, Azure Key Vault, Azure DevOps, and Azure Synapse.
• Build secure data pipelines using Unity Catalog, RBAC, service principals, and managed identities.
• Develop reusable notebook frameworks using PySpark and Spark SQL.
• Manage environment promotion across Development, QA, UAT, and Production.
• Troubleshoot production issues and optimize workloads for reliability and scalability.

Data Integration & Analytics

• Design enterprise data models supporting reporting, analytics, and downstream applications.
• Develop healthcare and financial data integration pipelines supporting multiple source systems.
• Integrate third-party APIs including NLP, terminology normalization, and identity resolution services.
• Support data governance, lineage, and metadata management initiatives.

Required Skills & Experience

Technical Expertise

• 10-12 years of experience in Data Engineering, ETL Development, and Data Warehousing.
• Strong experience with Spark, Delta Lake, Unity Catalog, Databricks Workflows, and SQL Warehouses.
• Strong experience with PySpark, Spark SQL, SQL, and Python.
• Experience building enterprise ETL/ELT pipelines using Databricks and Azure Data Platform.
• Experience with Azure Data Lake Storage (ADLS Gen2), Azure Synapse, Azure Key Vault, and Azure DevOps.
• Experience implementing CDC, SCD, incremental loading, and data quality frameworks.
• Experience integrating REST APIs and processing JSON/XML datasets.
• Experience with Git, CI/CD, release management, and deployment automation.
• Strong knowledge of performance tuning, partitioning, caching, broadcast joins, and Spark optimization.
• Experience working with healthcare data platforms is preferred.
• Microsoft Azure certifications (e.g., DP-203 Azure Data Engineer Associate) or Databricks certifications (Data Engineer Associate/Professional).
• Experience with Ab Initio suite of products is also preferred.

Base Compensation Range: 63,000 - 86,500

The posted range is the hiring range for this role — a subset of the broader range available to employees over time — and reflects base salary across our national hiring scale. Final offers are based on several factors, including the candidate's skills and experience, internal pay equity, work location, market conditions for the role, and the specific scope and responsibilities of the position. The top of the range is reserved for candidates who notably exceed the requirements; the lower end applies to those with less experience or fewer preferred qualifications. For positions based in higher-cost zones (e.g., California, New York, New Jersey), actual compensation may exceed the posted range; your recruiter will share specifics during the process.

For more information on benefits and what we offer please visit us at https://www.exlservice.com/us-careers-and-benefits