Azure Data Engineer

Advance Career Solutions; Pune District; On-site; INR 4,000,000 - 6,000,000; Full time

Job summary

Advance Career Solutions in Pune, Maharashtra, seeks an experienced Azure Data Engineer with 8+ years of experience to design, develop, and implement scalable data engineering solutions on Microsoft Azure.

You will work with Azure Databricks, Azure Data Factory, SQL and cloud-based ETL/ELT pipelines to build high-performance data platforms, optimize processing, and deliver enterprise-grade analytics.

Qualifications

• 8+ years of experience in Data Engineering and ETL development.
• Strong experience with Azure Data Factory (ADF).
• Strong experience with Azure Databricks.
• Hands‑on experience with PySpark and Apache Spark.
• Advanced SQL development and query optimization.
• Experience with Azure Data Lake Storage (ADLS Gen2).
• Knowledge of Delta Lake architecture.
• Experience in designing cloud-based data pipelines.
• Strong understanding of data warehousing concepts, including Star and Snowflake schemas.
• Experience with Git and Azure DevOps.
• Familiarity with Agile/Scrum methodologies.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Azure Data Factory (ADF).
• Develop and optimize ETL/ELT solutions using Azure Databricks and PySpark.
• Create reusable notebooks and workflows for large-scale data processing.
• Design and implement Delta Lake architecture and optimize Spark workloads.
• Develop complex SQL queries, stored procedures, views, and performance tuning techniques.
• Integrate data from multiple structured and unstructured data sources.
• Build data ingestion frameworks using Azure Data Lake Storage (ADLS Gen2).
• Perform data validation, reconciliation, and quality checks.
• Collaborate with business analysts, architects, and data scientists to understand business requirements.
• Implement monitoring, logging, and error-handling mechanisms for data pipelines.
• Optimize Azure resources for performance, scalability, and cost efficiency.
• Participate in code reviews, design discussions, and production support activities.
• Follow DevOps best practices using Git and CI/CD pipelines.
• Ensure compliance with data security, governance, and organizational standards.
• Mentor junior team members and provide technical guidance.

Skills

8+ years of experience in Data Eng

Azure Data Factory

Azure Databricks

PySpark

SQL

ADLS Gen2

Delta Lake

Spark

Git

Azure DevOps

Agile/Scrum

Tools

Azure Databricks

Azure Data Factory

Azure Data Lake Storage Gen2

Delta Lake

Git

Azure DevOps

PySpark

Job description

Role & responsibilities

Job Summary

We are seeking an experienced Azure Data Engineer with over 8 years of experience in designing, developing, and implementing scalable data engineering solutions on Microsoft Azure. The ideal candidate should have strong expertise in Azure Databricks, Azure Data Factory, SQL, and cloud-based ETL/ELT pipelines. The role involves building high-performance data platforms, optimizing data processing, and collaborating with cross-functional teams to deliver enterprise-grade analytics solutions.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Azure Data Factory (ADF).
• Develop and optimize ETL/ELT solutions using Azure Databricks and PySpark.
• Create reusable notebooks and workflows for large-scale data processing.
• Design and implement Delta Lake architecture and optimize Spark workloads.
• Develop complex SQL queries, stored procedures, views, and performance tuning techniques.
• Integrate data from multiple structured and unstructured data sources.
• Build data ingestion frameworks using Azure Data Lake Storage (ADLS Gen2).
• Perform data validation, reconciliation, and quality checks.
• Collaborate with business analysts, architects, and data scientists to understand business requirements.
• Implement monitoring, logging, and error-handling mechanisms for data pipelines.
• Optimize Azure resources for performance, scalability, and cost efficiency.
• Participate in code reviews, design discussions, and production support activities.
• Follow DevOps best practices using Git and CI/CD pipelines.
• Ensure compliance with data security, governance, and organizational standards.
• Mentor junior team members and provide technical guidance.

Required Skills

• 8+ years of experience in Data Engineering and ETL development.
• Strong experience with Azure Data Factory (ADF).
• Strong experience with Azure Databricks.
• Hands‑on experience with PySpark and Apache Spark.
• Advanced SQL development and query optimization.
• Experience with Azure Data Lake Storage (ADLS Gen2).
• Knowledge of Delta Lake architecture.
• Experience in designing cloud-based data pipelines.
• Strong understanding of data warehousing concepts, including Star and Snowflake schemas.
• Experience with Git and Azure DevOps.
• Familiarity with Agile/Scrum methodologies.
• Excellent analytical, troubleshooting, and communication skills.

Preferred candidate profile