Azure Data Engineer

Infosys; Bengaluru; On-site; INR 1,200,000 - 2,100,000; Full time

Job summary

Infosys in Bengaluru is seeking an experienced Data Engineer to design and maintain end-to-end data pipelines using Azure Data Factory, Databricks, and Synapse. You will implement robust monitoring, data validation, and performance optimizations to meet business needs.

The role requires hands-on ADF and Databricks experience, collaboration with cross-functional teams, and adherence to release and change management practices. 3–5 years of data engineering experience are expected.

Qualifications

• Education: BTECH, MTECH, MCA, MSC.
• Overall experience: 3-5 years in data engineering / data integration roles.
• Hands-on experience with Azure Data Factory including pipeline development, scheduling, triggers, and integration patterns.
• Hands-on experience with Databricks for data processing and transformation workflows.
• Ability to troubleshoot pipeline failures, analyze logs, and implement corrective actions to improve stability.

Responsibilities

• Design, develop, and maintain end-to-end data pipelines using Azure Data Factory for batch and scheduled workloads.
• Build and orchestrate data transformations and processing workflows using Databricks aligned to business requirements.
• Implement robust pipeline monitoring, alerting, and failure handling to ensure reliable and repeatable executions.
• Perform data validation and reconciliation checks to ensure accuracy, completeness, and consistency across sources and targets.
• Optimize pipeline performance by tuning ADF activities, improving orchestration logic, and streamlining transformations in Databricks.
• Collaborate with cross-functional teams to gather requirements, estimate effort, and deliver solutions within timelines.
• Maintain technical documentation for pipelines, datasets, schedules, dependencies, and operational runbooks.
• Support deployments and environment promotions by following structured release and change management practices.

Skills

Azure Data Factory

Azure Synapse Analytics

Azure Data Lake Storage

PySpark

Delta Lake

Databricks

CI/CD for data pipelines

Education

BTECH

MTECH

MCA

MSC

Tools

Databricks

Azure Data Factory (ADF)

Job description

Azure Datafactory Azure Synapse Analytics, Azure Data Lake Storage (ADLS), PySpark, Delta Lake, CI/CD for data pipelines, AZURE DATAFACTORY

• Experience designing scalable orchestration patterns in ADF (parameterization, reusable components, dependency handling).
• Strong understanding of data transformation best practices and implementing efficient processing in Databricks.
• Exposure to building data quality checks and operational dashboards for pipeline health and SLA tracking.
• Experience collaborating with stakeholders to translate business requirements into technical pipeline designs and delivery plans.
• Familiarity with performance tuning approaches for cloud data pipelines and distributed processing workloads.

Key Responsibilities:

• Design, develop, and maintain end-to-end data pipelines using Azure Data Factory for batch and scheduled workloads.
• Build and orchestrate data transformations and processing workflows using Databricks aligned to business requirements.
• Implement robust pipeline monitoring, alerting, and failure handling to ensure reliable and repeatable executions.
• Perform data validation and reconciliation checks to ensure accuracy, completeness, and consistency across sources and targets.
• Optimize pipeline performance by tuning ADF activities, improving orchestration logic, and streamlining transformations in Databricks.
• Collaborate with cross-functional teams to gather requirements, estimate effort, and deliver solutions within timelines.
• Maintain technical documentation for pipelines, datasets, schedules, dependencies, and operational runbooks.
• Support deployments and environment promotions by following structured release and change management practices.

Minimum Qualifications:

• Education: BTECH, MTECH, MCA, MSC.
• Overall experience: 3-5 years in data engineering / data integration roles.
• Hands-on experience with Azure Data Factory including pipeline development, scheduling, triggers, and integration patterns.
• Hands-on experience with Databricks for data processing and transformation workflows.
• Ability to troubleshoot pipeline failures, analyze logs, and implement corrective actions to improve stability.