Azure Data Engineer

Epergne Solutions; Chennai District; On-site; INR 1,800,000 - 2,400,000; Full time

Job summary

A technology solutions provider in Chennai is seeking an experienced Azure Data Engineer to design and maintain scalable ETL/ELT pipelines. The ideal candidate will have hands-on experience with Python, PySpark, and Azure Databricks, along with strong knowledge of Delta Lake architecture. Responsibilities include implementing data validation and monitoring pipelines. This role offers exciting opportunities within a collaborative environment focused on modern data engineering practices.

Qualifications

• 5+ years of experience in data engineering roles.
• Strong hands-on experience in PySpark and Spark SQL.
• Proven experience in Spark performance tuning and optimization.
• Experience working with Azure Data Lake Storage Gen2.
• Git-based version control experience.
• Production support and troubleshooting experience.

Responsibilities

• Design, develop, and maintain ETL/ELT pipelines.
• Build and optimize workloads on Azure Databricks.
• Implement Delta Lake best practices and ensure performance optimization.
• Develop robust ingestion frameworks integrating with Azure Data Lake Storage Gen2 and Azure SQL.
• Implement data validation, transformation logic, and quality control frameworks.
• Ensure performance optimization and cost-efficient execution.
• Support production deployments, monitor pipelines, and resolve incidents.
• Maintain version control using Git and follow CI/CD best practices.
• Collaborate with architects, analysts, and business stakeholders.

Skills

Python

PySpark

Spark SQL

Azure Databricks

Data Lake Storage Gen2

Git

CI/CD

CI/CD

Spark performance tuning

Delta Lake

Azure Data Lake Storage Gen2

Tools

Azure Data Factory

SAP ERP systems

SAP ERP systems

Job description

Experience: 5+ Years

We are looking for a highly skilled Azure Data Engineer for an engagement to build scalable, production-ready data solutions.

The ideal candidate will have strong hands?on experience with Python, PySpark, Azure Databricks, and Azure Data Factory, along with a solid understanding of modern data engineering best practices.

Key Responsibilities

Design, develop, and maintain scalable ETL/ELT pipelines using Python, PySpark, and Spark SQL

Build and optimize workloads on Azure Databricks (notebooks, job clusters, workflows)

Implement Delta Lake best practices (schema evolution, partitioning, performance tuning, merge strategies)

• Develop robust ingestion frameworks integrating with Azure Data Lake Storage Gen2 and Azure SQL
• Implement data validation, transformation logic, and quality control frameworks
• Ensure performance optimization and cost-efficient execution
• Support production deployments, monitor pipelines, and resolve incidents
• Maintain version control using Git and follow CI/CD best practices
• Collaborate with architects, analysts, and business stakeholders

Mandatory Skills

Strong hands?on experience in PySpark and Spark SQL

Proven experience in Spark performance tuning & optimization

Strong knowledge of Delta Lake architecture

• Experience working with Azure Data Lake Storage Gen2
• Git-based version control experience
• Production support and troubleshooting experience

Good to Have

Exposure to data governance, data security, and compliance frameworks

• Experience integrating data from SAP ERP systems
• Knowledge of Azure Data Factory (ADF)
• Understanding of DevOps and CI/CD pipelines