Lead Data Engineer

IRT Digital Analytics Solutions; Mumbai Suburban; On-site; INR 1,000,000 - 1,500,000; Full time

Job summary

IRT Digital Analytics Solutions is seeking a skilled Databricks Developer to design and build scalable data pipelines using Azure Databricks and PySpark. The role focuses on data transformation, integration, and lakehouse architectures to enable trusted analytics across teams.

You will work with Python, SQL, Spark, and Delta Lake, optimize pipelines for performance and cost, and implement robust data security and governance.

Qualifications

• Hands-on experience with Azure Databricks and PySpark.
• Strong knowledge of Python and SQL.
• Experience with Apache Spark, Delta Lake, and ETL/ELT pipelines.
• Understanding data warehousing and lakehouse architecture.
• Experience with Azure Data Factory, ADLS, or other cloud data services.
• Knowledge of Git, CI/CD, and workflow orchestration.
• Strong analytical and problem-solving skills.

Responsibilities

• Design, develop, and maintain data pipelines using Azure Databricks and PySpark.
• Develop ETL/ELT processes for data ingestion, transformation, and integration.
• Work with Python, SQL, Apache Spark, and Delta Lake.
• Implement data lakehouse solutions and optimize data processing performance.
• Integrate data from multiple sources and ensure data quality.
• Collaborate with data engineers, architects, and business teams to deliver data solutions.
• Troubleshoot pipeline failures and optimize jobs for performance and cost.
• Implement data security, governance, and access controls.

Skills

Databricks PySpark

Python

SQL

Apache Spark

Delta Lake

ETL/ELT pipelines

Data warehousing

Lakehouse

Azure Data Factory

ADLS

Git

CI/CD

Orchestration

Analytical skills

Tools

Unity Catalog

Delta Live Tables

Azure cloud services

Job description

Job Summary

We are looking for a skilled Databricks Developer with experience in data engineering, data transformation, and building scalable data pipelines using Azure Databricks and Apache Spark.

Key Responsibilities

• Design, develop, and maintain data pipelines using Azure Databricks and PySpark.
• Develop ETL/ELT processes for data ingestion, transformation, and integration.
• Work with Python, SQL, Apache Spark, and Delta Lake.
• Implement data lakehouse solutions and optimize data processing performance.
• Integrate data from multiple sources and ensure data quality.
• Collaborate with data engineers, architects, and business teams to deliver data solutions.
• Troubleshoot pipeline failures and optimize jobs for performance and cost.
• Implement data security, governance, and access controls.

Required Skills

• Hands-on experience with Azure Databricks and PySpark.
• Strong knowledge of Python and SQL.
• Experience with Apache Spark, Delta Lake, and ETL/ELT pipelines.
• Understanding data warehousing and lakehouse architecture.
• Experience with Azure Data Factory, ADLS, or other cloud data services.
• Knowledge of Git, CI/CD, and workflow orchestration.
• Strong analytical and problem-solving skills.

Preferred Qualifications

• Experience with Databricks Unity Catalog and Delta Live Tables (Lakeflow Declarative Pipelines).
• Knowledge of Azure cloud services and data governance.
• Experience in performance tuning and production deployment of data pipelines.