Lead Data Engineer

Experis; Pune District; On-site; INR 2,500,000 - 5,000,000; Full time

Job summary

Experis in Pune is seeking an experienced Lead Data Engineer to design, build, and maintain scalable data pipelines on the Databricks platform. You will modernize legacy ETL processes, optimize Spark jobs, and drive end-to-end data workflows from ingestion to consumption.

The ideal candidate has 8+ years of experience and deep expertise in Databricks, PySpark, Python, SQL, Delta Lake, and data modeling, with strong focus on CI/CD and Agile development. This is a full-time on-site role in Pune.

Qualifications

• 8+ years of experience in data engineering or related roles.
• Proficiency with Databricks, PySpark, Python, SQL and ETL.
• Experience designing scalable data pipelines and modernizing legacy ETL processes.

Responsibilities

• Design, build, and maintain scalable data pipelines on the Databricks platform.
• Develop end-to-end data workflows from ingestion to consumption.
• Refactor legacy ETL to PySpark and modern ELT patterns.
• Optimize Spark jobs, clusters, and Delta Lake tables for performance.
• Implement data quality checks, monitoring, and error handling.
• Manage Databricks Jobs and workflow orchestration.
• Follow software engineering best practices, including version control, testing, and CI/CD.
• Mentor junior engineers and maintain technical documentation.

Skills

Databricks

PySpark

Python

SQL

ETL

Data Pipeline Development

Delta Lake

Data Modelling

Databricks Jobs & Workflows

Legacy ETL Modernization

Git

CI/CD

Agile Development

Databricks Certified Data Engineer Ass

Job description

Position: Lead Data Engineer

Employment Type: Full-Time

Experience: 8+ Years

Location: Pune

Job Summary

We are looking for an experienced Lead Data Engineer with strong expertise in Databricks, PySpark, Python, SQL, and ETL. The candidate will be responsible for designing, building, and maintaining scalable data pipelines, modernizing legacy ETL processes, and delivering high-performance data engineering solutions on the Databricks platform.

Mandatory Skills

• Databricks
• PySpark
• Python
• SQL
• ETL
• Data Pipeline Development
• Delta Lake
• Data Modelling
• Databricks Jobs & Workflows
• Legacy ETL Modernization
• Git
• CI/CD
• Agile Development
• Databricks Certified Data Engineer Associate OR Databricks Certified Data Engineer Professional

Good to Have

• Azure, AWS, or GCP
• Structured Streaming
• Workspace AI Agent
• Data Governance and Security
• DevOps Practices

Key Responsibilities

• Design, build, and maintain scalable data pipelines on the Databricks platform.
• Develop end-to-end data workflows from ingestion to transformation and consumption.
• Refactor legacy code and ETL pipelines to PySpark and modern ELT patterns.
• Optimize Spark jobs, clusters, and Delta Lake tables for performance.
• Implement data quality checks, monitoring, and error handling.
• Manage Databricks Jobs and workflow orchestration.
• Follow software engineering best practices, including version control, testing, and CI/CD.
• Troubleshoot and resolve production data pipeline issues.
• Collaborate with architects, analysts, infrastructure, application, and cyber teams.
• Mentor junior engineers and maintain technical documentation.

Required Certification (Mandatory)

• Databricks Certified Data Engineer Associate OR
• Databricks Certified Data Engineer Professional

Preferred Certifications

• Databricks Certified Associate Developer for Apache Spark
• Azure Data Engineer Associate
• AWS Certified Data Analytics
• Google Cloud Professional Data Engineer