Senior Data Engineer (Databricks & PySpark)

Experis; Pune District; On-site; INR 4,000,000 - 7,000,000; Full time

Job summary

Experis in Pune is seeking a Senior Data Engineer to design, develop, and maintain scalable data pipelines on the Databricks platform. You will modernize legacy ETL processes, implement Delta Lake, optimize Spark workloads, and collaborate with cross-functional teams.

The role emphasizes writing reusable code, adhering to CI/CD practices, and mentoring junior engineers on PySpark and Databricks best practices.

Qualifications

• Experience designing scalable data pipelines on Databricks.
• Strong PySpark and Python programming skills.
• Familiarity with Delta Lake and data quality checks.

Responsibilities

• Design, build, and operate scalable data pipelines on the Databricks platform.
• Migrate ETL to ELT on Databricks and modernize legacy code.
• Implement monitoring, alerting, and error-handling for pipelines.
• Collaborate with data architects, analysts, and stakeholders.
• Mentor junior engineers on Databricks and PySpark.

Skills

Databricks

PySpark

Python

Spark

Data pipelines

Tools

Databricks

Delta Lake

CI/CD

Git

Job description

Job Overview

We are looking for a Senior Data Engineer with strong experience in Databricks, PySpark, and Python to design, develop, and maintain scalable data pipelines. The role involves modernizing legacy ETL processes, implementing Delta Lake solutions, optimizing Spark workloads, and collaborating with cross-functional teams to deliver reliable data engineering solutions.

Key Responsibilities

Data Pipeline Development & Operations

• Design, build, and operate scalable data pipelines on the Databricks platform.
• Develop end-to-end data workflows from ingestion to transformation and consumption.
• Implement monitoring, alerting, and error-handling mechanisms.
• Optimize Spark jobs and cluster configurations.
• Manage Databricks Jobs and workflow orchestration.

Legacy Code Modernization

• Refactor legacy data pipelines into PySpark.
• Migrate ETL processes to ELT on Databricks.
• Assess existing codebases for optimization opportunities.
• Ensure data integrity and backward compatibility.
• Document migration approaches and playbooks.

Data Engineering

• Implement data quality checks and validation frameworks.
• Design and maintain Delta Lake tables.
• Develop reusable code libraries.
• Follow version control, testing, and CI/CD practices.
• Participate in code reviews.
• Troubleshoot production data pipelines.

Collaboration

• Work with data architects, analysts, business stakeholders, Infrastructure, Applications, and Cyber teams.
• Mentor junior engineers on Databricks and PySpark.
• Share technical knowledge and maintain documentation.