Data Engineer with Data Bricks

Exponentia.ai Private Limited; Mumbai City; Hybrid; INR 1,500,000 - 2,300,000; Full time

Job summary

Exponentia.ai Private Limited is seeking a Senior Databricks Data Engineer to design and develop scalable data pipelines on Azure Databricks. You will optimize Spark workloads and migrate Finance data from on-prem SAP systems into a modern cloud Lakehouse.

The role emphasizes building Bronze/Silver/Gold layers, implementing Delta Lake practices, and collaborating with BI teams for optimized reporting datasets.

Qualifications

• 5+ years of experience in data engineering with Spark-based workloads.
• Proficient in PySpark and Spark SQL.
• Experience building Bronze/Silver/Gold data layers in a lakehouse architecture.
• Solid understanding of Delta Lake best practices.
• Experience with SAP data integration and on-prem-to-cloud migrations.

Responsibilities

• Design and develop scalable ETL/ELT pipelines.
• Develop PySpark and Spark SQL transformations.
• Build Bronze, Silver, and Gold layer data pipelines.
• Integrate SAP and enterprise source systems.
• Implement Delta Lake best practices.
• Optimize Spark jobs for performance and cost.
• Build reusable data engineering frameworks.
• Implement data quality and validation rules.
• Develop incremental and CDC pipelines.
• Collaborate with BI developers for optimized reporting datasets.
• Participate in code reviews and technical documentation.

Skills

PySpark

Spark SQL

Delta Lake

ETL/ELT

Data pipelines

CDC pipelines

Azure Databricks

SAP integration

Data quality

Code reviews

Tools

Azure Databricks

Delta Lake

Job description

About the Role:

We are seeking a Senior Databricks Data Engineer to design and develop scalable data pipelines, optimize Spark workloads, and build enterprise-grade data engineering solutions using Azure Databricks. The engineer will play a critical role in migrating Finance data from on-premises SAP systems into a modern cloud Lakehouse architecture.

Key Responsibilities

• Design and develop scalable ETL/ELT pipelines.


• Develop PySpark and Spark SQL transformations.


• Build Bronze, Silver, and Gold layer data pipelines.


• Integrate SAP and enterprise source systems.


• Implement Delta Lake best practices.


• Optimize Spark jobs for performance and cost.


• Build reusable data engineering frameworks.


• Implement data quality and validation rules.


• Develop incremental and CDC pipelines.


• Collaborate with BI developers for optimized reporting datasets.


• Participate in code reviews and technical documentation.


• Support production deployments and troubleshooting.