Senior Azure Databricks Data Engineer - Scale AI Pipelines

EXL; New York (NY); Hybrid; USD 140,000 - 190,000; Full time

Job summary

EXL is seeking a Senior Azure Databricks Data Engineer to design, develop, and optimize scalable data platforms using the Databricks Lakehouse Platform. The role emphasizes data engineering, ETL/ELT development, and cloud-native architectures to support enterprise analytics and AI initiatives.

The candidate should have 9–15 years IT experience, strong Spark/PySpark skills, and hands-on Azure data engineering expertise, with a focus on data quality, governance, and performance optimization.

Qualifications

• 9–15 years of IT experience in Data Engineering.
• 5+ years of hands-on Azure Data Engineering technologies.
• 4+ years of hands-on experience with Azure Databricks and Spark ecosystem.
• Strong understanding of data lake, lakehouse, and cloud-native architecture patterns.
• Experience handling large-scale structured and unstructured datasets.

Responsibilities

• Design, develop, and maintain data pipelines using Databricks, Apache Spark, and cloud-native services.
• Build and optimize ETL/ELT workflows for large-scale data.
• Develop data models and implement data quality, validation, and governance frameworks.
• Integrate data from multiple sources into a unified Lakehouse architecture.
• Optimize Spark jobs and Databricks workloads for performance, scalability, and cost efficiency.
• Implement security controls, access management, and data governance using Unity Catalog.
• Collaborate with business, analytics, and AI/ML teams to deliver trusted data products.
• Monitor, troubleshoot, and resolve data pipeline issues.
• Support CI/CD, DevOps, and infrastructure automation practices.
• Maintain technical documentation and best practices.

Skills

Databricks Lakehouse Platform

Apache Spark / PySpark

Delta Lake

SQL

Python

Education

Bachelor's or Master's degree in Computer Science/IT/Engineering

Tools

Terraform

GitHub Actions

Azure DevOps

CI/CD Pipelines

Job description

EXL is seeking a Senior Azure Databricks Data Engineer to design, develop, and optimize scalable data platforms using the Databricks Lakehouse Platform. The role emphasizes data engineering, ETL/ELT development, and cloud-native architectures to support enterprise analytics and AI initiatives.

The candidate should have 9–15 years IT experience, strong Spark/PySpark skills, and hands-on Azure data engineering expertise, with a focus on data quality, governance, and performance optimization.