Senior Databricks Data Engineer

Iitjobs Inc.; Mumbai; Hybrid; INR 4,800,000 - 7,000,000; Full time

Job summary

Iitjobs Inc. in Mumbai seeks a Senior Databricks Data Engineer to design scalable data pipelines and optimize Spark workloads, migrating Finance data from on-prem SAP systems to a cloud Lakehouse.

You will implement Bronze/Silver/Gold pipelines, Delta Lake best practices, and data quality rules. The ideal candidate will have hands-on Azure Databricks, PySpark, Spark SQL expertise, and experience with ETL/ELT, ADLS Gen2, and Azure Data Factory.

Qualifications

• 812 years of Data Engineering experience.
• Strong hands-on experience with Azure Databricks.
• Expert in PySpark and Spark SQL.
• Experience with Delta Lake.
• Experience with Azure Data Factory.
• Knowledge of ADLS Gen2.
• Strong SQL skills.
• Experience with Git and CI/CD.
• Understanding of SAP data extraction techniques.

Responsibilities

• Design and develop scalable ETL/ELT pipelines.
• Develop PySpark and Spark SQL transformations.
• Build Bronze, Silver, and Gold layer data pipelines.
• Integrate SAP and enterprise source systems.
• Implement Delta Lake best practices.
• Optimize Spark jobs for performance and cost.
• Build reusable data engineering frameworks.
• Implement data quality and validation rules.
• Develop incremental and CDC pipelines.
• Collaborate with BI developers for optimized reporting datasets.
• Participate in code reviews and technical documentation.
• Support production deployments and troubleshooting.

Skills

PySpark

Spark SQL

SQL

SAP data extraction techniques

Tools

Azure Databricks

Delta Lake

Azure Data Factory

ADLS Gen2

Git

CI/CD

Power BI datasets

Job description

About the Role:

We are seeking a Senior Databricks Data Engineer to design and develop scalable data pipelines, optimize Spark workloads, and build enterprise-grade data engineering solutions using Azure Databricks. The engineer will play a critical role in migrating Finance data from on-premises SAP systems into a modern cloud Lakehouse architecture.

Key Responsibilities

• Design and develop scalable ETL/ELT pipelines.
• Develop PySpark and Spark SQL transformations.
• Build Bronze, Silver, and Gold layer data pipelines.
• Integrate SAP and enterprise source systems.
• Implement Delta Lake best practices.
• Optimize Spark jobs for performance and cost.
• Build reusable data engineering frameworks.
• Implement data quality and validation rules.
• Develop incremental and CDC pipelines.
• Collaborate with BI developers for optimized reporting datasets.
• Participate in code reviews and technical documentation.
• Support production deployments and troubleshooting.

Roles & Responsibilities

Ideal Candidate

• 812 years of Data Engineering experience.
• Strong hands-on experience with Azure Databricks.
• Expert in PySpark and Spark SQL.
• Experience with Delta Lake.
• Experience with Azure Data Factory.
• Knowledge of ADLS Gen2.
• Strong SQL skills.
• Experience with Git and CI/CD.
• Understanding of SAP data extraction techniques.

Preferred Certification

• Databricks Certified Data Engineer Professional.
• Experience with Structured Streaming.
• Knowledge of MLflow.
• Azure DevOps experience.
• Experience with Power BI datasets.