Senior Azure Databricks Data Engineer

EXL Talent Acquisition Team; New York (NY); Hybrid; USD 140,000 - 190,000; Full time

Job summary

EXL Talent Acquisition Team in the United States is seeking a Senior Azure Databricks Data Engineer to design, develop, and optimize scalable data platforms using the Databricks Lakehouse Platform, with a focus on Azure cloud technologies and big data processing for enterprise analytics.

The role requires 9-15 years IT experience, hands-on Databricks, Spark, Delta Lake, SQL and Python, plus cloud and DevOps proficiency to support AI initiatives.

Qualifications

• Bachelor's or Master's degree in CS/IT/Engineering or related field.
• 9-15 years of experience in Data Engineering and Data Warehousing.
• Minimum 5+ years of hands-on experience with Azure Data Engineering technologies.
• Minimum 4+ years of hands-on experience with Azure Databricks and Spark ecosystem.
• Strong understanding of data lake, lakehouse, and cloud-native architecture patterns.
• Experience in handling large-scale structured and unstructured datasets.

Responsibilities

• Design, develop, and maintain data pipelines using Databricks, Apache Spark, and cloud-native services.

Skills

Databricks Lakehouse Platform

Apache Spark / PySpark

Delta Lake

SQL

Python

Education

Bachelor's or Master's degree in CS/IT/Engineering or related field

Tools

Azure

Terraform

GitHub Actions

Azure DevOps

Job description

Job Summary
We are seeking a skilled Databricks Engineer with minimum of 5+ years of hands-on in Databricks who can design, develop, and optimize scalable data platforms and analytics solutions using the Databricks Lakehouse Platform. The ideal candidate will have expertise in data engineering, ETL/ELT development, cloud technologies preferably Azure, and big data processing to support enterprise analytics and AI initiatives.

Job Description: Azure / Databricks Data Engineer (9-15 Years Experience)

Senior Azure Databricks Data Engineer

Experience

9-15 Years of IT Experience

Location

Hybrid

Job Summary

We are seeking a skilled Databricks Engineerwith minimum of 5+ years of hands-on in Databrickswho can design, develop, and optimize scalable data platforms and analytics solutions using the Databricks Lakehouse Platform. The ideal candidate will have expertise in data engineering, ETL/ELT development, cloud technologies preferably Azure, and big data processing to support enterprise analytics and AI initiatives.

Key Responsibilities

• Design, develop, and maintain data pipelines using Databricks, Apache Spark, and cloud-native services.


• Build and optimize ETL/ELT workflows for large-scale structured and unstructured data.


• Develop data models and implement data quality, validation, and governance frameworks.


• Integrate data from multiple sources into a unified Lakehouse architecture.


• Optimize Spark jobs and Databricks workloads for performance, scalability, and cost efficiency.


• Implement security controls, access management, and data governance using Unity Catalog.


• Collaborate with business, analytics, and AI/ML teams to deliver trusted data products.


• Monitor, troubleshoot, and resolve data pipeline issues.


• Support CI/CD, DevOps, and infrastructure automation practices.


• Maintain technical documentation and best practices.

Required Technical Skills Core Technologies

• Databricks Lakehouse Platform


• Apache Spark / PySpark


• Delta Lake


• SQL


• Python

Data Engineering

• ETL / ELT Development


• Data Modeling


• Data Warehousing


• Data Quality & Validation


• Streaming & Real-Time Processing

Governance & Security

• Unity Catalog


• Data Lineage


• Row-Level Security


• Access Control & Compliance


• Data Governance Frameworks

Cloud & DevOps

• Azure / AWS / GCP


• Terraform


• GitHub Actions / Azure DevOps


• CI/CD Pipelines

Analytics & AI

• Semantic Layers


• Data Products


• BI Platforms


• Machine Learning Support


• Generative AI & RAG Architectures

Required Qualifications

• Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or related field.


• 9-15 years of experience in Data Engineering and Data Warehousing.


• Minimum 5+ years of hands-on experience with Azure Data Engineering technologies.


• Minimum 4+ years of hands-on experience with Azure Databricks and Spark ecosystem.


• Strong understanding of data lake, lakehouse, and cloud-native architecture patterns.


• Experience in handling large-scale structured and unstructured datasets.


• Strong analytical, problem-solving, and troubleshooting skills.

Preferred Qualifications

• Microsoft Certified: Azure Data Engineer Associate (DP-203).


• Databricks Certified Data Engineer Associate/Professional.


• Experience with Snowflake, Power BI, or Microsoft Fabric.


• Experience in real-time streaming solutions using Kafka/Event Hubs.


• Exposure to Data Governance and Master Data Management initiatives.

Soft Skills

• Strong stakeholder management and communication skills.


• Ability to lead technical initiatives and drive architecture discussions.


• Experience working in Agile/Scrum environments.


• Excellent documentation and presentation skills.


• Strong mentoring and team leadership abilities.

Nice to Have

• Microsoft Fabric


• Power BI


• Azure Event Hubs


• Kafka


• Machine Learning data pipelines


• Data Governance tools such as Purview