Lead Data Engineer

Mployee.me; Mumbai; Presencial; INR 1.500.000 - 2.100.000; Jornada completa

Descripción de la vacante

Mployee.me in Mumbai, India, is seeking a skilled Databricks Developer to architect scalable data pipelines using Azure Databricks and Apache Spark. You will drive data ingestion, transformation, and integration across sources, leveraging Python, SQL, and Delta Lake to deliver robust data lakehouse solutions.

You will collaborate with data engineers, architects, and business teams to ensure data quality, security, and efficient, cost-optimized processing in a cloud-first environment.

Formación

• Hands-on experience with Azure Databricks and PySpark.
• Strong knowledge of Python and SQL.
• Experience with Apache Spark, Delta Lake, and ETL/ELT pipelines.
• Understanding data warehousing and lakehouse architecture.
• Experience with Azure Data Factory, ADLS, or other cloud data services.
• Knowledge of Git, CI/CD, and workflow orchestration.
• Strong analytical and problem-solving skills.

Responsabilidades

• Design, develop, and maintain data pipelines using Azure Databricks and PySpark.
• Develop ETL/ELT processes for data ingestion, transformation, and integration.
• Work with Python, SQL, Apache Spark, and Delta Lake.
• Implement data lakehouse solutions and optimize data processing performance.
• Integrate data from multiple sources and ensure data quality.
• Collaborate with data engineers, architects, and business teams to deliver data solutions.
• Troubleshoot pipeline failures and optimize jobs for performance and cost.
• Implement data security, governance, and access controls.

Conocimientos

Azure Databricks

PySpark

Python

SQL

Apache Spark

Delta Lake

ETL/ELT pipelines

Data governance

CI/CD

Workflow orchestration

Data warehousing/lakehouse

Herramientas

Azure Data Factory

ADLS

Git

Descripción del empleo

Job Summary

We are looking for a skilled Databricks Developer with experience in data engineering, data transformation, and building scalable data pipelines using Azure Databricks and Apache Spark.

Key Responsibilities

• Design, develop, and maintain data pipelines using Azure Databricks and PySpark.
• Develop ETL/ELT processes for data ingestion, transformation, and integration.
• Work with Python, SQL, Apache Spark, and Delta Lake.
• Implement data lakehouse solutions and optimize data processing performance.
• Integrate data from multiple sources and ensure data quality.
• Collaborate with data engineers, architects, and business teams to deliver data solutions.
• Troubleshoot pipeline failures and optimize jobs for performance and cost.
• Implement data security, governance, and access controls.

Required Skills

• Hands-on experience with Azure Databricks and PySpark.
• Strong knowledge of Python and SQL.
• Experience with Apache Spark, Delta Lake, and ETL/ELT pipelines.
• Understanding data warehousing and lakehouse architecture.
• Experience with Azure Data Factory, ADLS, or other cloud data services.
• Knowledge of Git, CI/CD, and workflow orchestration.
• Strong analytical and problem-solving skills.

Preferred Qualifications

• Experience with Databricks Unity Catalog and Delta Live Tables (Lakeflow Declarative Pipelines).
• Knowledge of Azure cloud services and data governance.
• Experience in performance tuning and production deployment of data pipelines.