Sr Databricks Engineer

EXL; Pune District; Hybrid; INR 3,000,000 - 5,500,000; Full time

Job summary

EXL is seeking a Senior Azure Databricks Data Engineer to design, develop, and optimize scalable data platforms and analytics solutions on the Databricks Lakehouse Platform. The role focuses on ETL/ELT development, data modeling, and governance, with strong Azure cloud experience.

The ideal candidate has 9-15 years IT experience, with at least 5+ years in Azure data engineering and 4+ years in Azure Databricks and Spark, to support enterprise analytics and AI initiatives.

Qualifications

• 9-15 years of experience in Data Engineering and Data Warehousing.
• Minimum 5+ years of hands-on experience with Azure Data Engineering technologies.
• Minimum 4+ years of hands-on experience with Azure Databricks and Spark ecosystem.
• Strong understanding of data lake, lakehouse, and cloud-native architecture patterns.
• Experience in handling large-scale structured and unstructured datasets.

Responsibilities

• Design, develop, and maintain data pipelines using Databricks, Apache Spark, and cloud-native services.
• Build and optimize ETL/ELT workflows for large-scale structured and unstructured data.
• Develop data models and implement data quality, validation, and governance frameworks.
• Integrate data from multiple sources into a unified Lakehouse architecture.
• Optimize Spark jobs and Databricks workloads for performance, scalability, and cost efficiency.
• Implement security controls, access management, and data governance using Unity Catalog.
• Collaborate with business, analytics, and AI/ML teams to deliver trusted data products.
• Monitor, troubleshoot, and resolve data pipeline issues.
• Support CI/CD, DevOps, and infrastructure automation practices.
• Maintain technical documentation and best practices.

Skills

Databricks Lakehouse Platform

Apache Spark / PySpark

Delta Lake

SQL

Python

ETL / ELT Development

Data Modeling

Data Warehousing

Data Quality & Validation

Streaming & Real-Time Processing

Azure Cloud

Azure DevOps

Terraform

GitHub Actions

CI/CD Pipelines

Unity Catalog

Data Governance

Big Data

Machine Learning Support

RAG Architectures

Education

Bachelor's degree in Computer Science, IT, Engineering, or related field

Master's degree in a related field

Job description

Job Title

Senior Azure Databricks Data Engineer

Experience

9-15 Years of IT Experience

Location

Hybrid

Job Summary

We are seeking a skilled Databricks Engineer with minimum of 5+ years of hands-on in Databricks who can design, develop, and optimize scalable data platforms and analytics solutions using the Databricks Lakehouse Platform. The ideal candidate will have expertise in data engineering, ETL/ELT development, cloud technologies preferably Azure, and big data processing to support enterprise analytics and AI initiatives.

Key Responsibilities

• Design, develop, and maintain data pipelines using Databricks, Apache Spark, and cloud-native services.
• Build and optimize ETL/ELT workflows for large-scale structured and unstructured data.
• Develop data models and implement data quality, validation, and governance frameworks.
• Integrate data from multiple sources into a unified Lakehouse architecture.
• Optimize Spark jobs and Databricks workloads for performance, scalability, and cost efficiency.
• Implement security controls, access management, and data governance using Unity Catalog.
• Collaborate with business, analytics, and AI/ML teams to deliver trusted data products.
• Monitor, troubleshoot, and resolve data pipeline issues.
• Support CI/CD, DevOps, and infrastructure automation practices.
• Maintain technical documentation and best practices.

Required Technical Skills

Core Technologies

• Databricks Lakehouse Platform
• Apache Spark / PySpark
• Delta Lake
• SQL
• Python

Data Engineering

• ETL / ELT Development
• Data Modeling
• Data Warehousing
• Data Quality & Validation
• Streaming & Real-Time Processing

Governance & Security

• Unity Catalog
• Data Lineage
• Row-Level Security
• Access Control & Compliance
• Data Governance Frameworks

Cloud & DevOps

• Azure / AWS / GCP
• Terraform
• GitHub Actions / Azure DevOps
• CI/CD Pipelines

Analytics & AI

• Semantic Layers
• Data Products
• BI Platforms
• Machine Learning Support
• Generative AI & RAG Architectures

Required Qualifications

• Bachelors or Masters degree in Computer Science, Information Technology, Engineering, or related field.
• 9-15 years of experience in Data Engineering and Data Warehousing.
• Minimum 5+ years of hands-on experience with Azure Data Engineering technologies.
• Minimum 4+ years of hands-on experience with Azure Databricks and Spark ecosystem.
• Strong understanding of data lake, lakehouse, and cloud-native architecture patterns.
• Experience in handling large-scale structured and unstructured datasets.
• Strong analytical, problem-solving, and troubleshooting skills.

Preferred Qualifications

• Microsoft Certified: Azure Data Engineer Associate (DP-203).
• Databricks Certified Data Engineer Associate/Professional.
• Experience with Snowflake, Power BI, or Microsoft Fabric.
• Experience in real-time streaming solutions using Kafka/Event Hubs.
• Exposure to Data Governance and Master Data Management initiatives.

Soft Skills

• Strong stakeholder management and communication skills.
• Ability to lead technical initiatives and drive architecture discussions.
• Experience working in Agile/Scrum environments.
• Excellent documentation and presentation skills.
• Strong mentoring and team leadership abilities.

Nice to Have

• Microsoft Fabric
• Power BI
• Azure Event Hubs
• Kafka
• Machine Learning data pipelines
• Data Governance tools such as Purview

About EXL

EXL (NASDAQ: EXLS) is a leading data analytics and digital operations and solutions company. We partner with clients using a data and AI-led approach to reinvent business models, drive better business outcomes and unlock growth with speed. EXL harnesses the power of data, analytics, AI, and deep industry knowledge to transform operations for the world’s leading corporations in industries including insurance, healthcare, banking and financial services, media and retail, among others. EXL was founded in 1999 with the core values of innovation, collaboration, excellence, integrity and respect. We are headquartered in New York and have more than 54,000 employees spanning six continents. For more information, visit www.exlservice.com.

EXL never requires or asks for fees/payments or credit card or bank details during any phase of the recruitment or hiring process and has not authorized any agencies or partners to collect any fee or payment from prospective candidates. EXL will only extend a job offer after a candidate has gone through a formal interview process with members of EXL's Human Resources team, as well as our hiring managers.

EXL is the indispensable partner for leading businesses in data-led industries such as insurance, banking and financial services, healthcare, retail and logistics. We bring a unique combination of data, advanced analytics, digital technology and industry expertise to help our clients turn data into insights, streamline operations, improve customer experience, and transform their business. Our partnerships with clients are built on a foundation of collaboration - and we have been chosen as a partner by nine of the top ten leading US insurance companies, nine of the top 20 global banks, and six of the top ten US health care payers. We function as one team to make your goals our goals, whether that’s unlocking the value of generative AI or embedding analytics into workflows that reduce risk or power your growth. Clients choose EXL as their transformation partner for many reasons. Our geographic diversity make talent all over the world instantly accessible. Digital accelerators enable unmatched speed-to-value, letting you realize results fast. Its our people that truly set us apart, though, including the 1,500 data scientists we have dedicated to our generative AI practice. And our more than twenty years of experience in delivering business services, garnering stellar client references, and maintaining a solid balance sheet are reassuring to our C-suite clients. Find out for yourself why clients, employees, and analysts think we’re some of the best in the business. Contact us to see how we can help you achieve your goals.