ADF (Azure Data Factory)Databricks+Pyspark

Infosys; Bengaluru; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

Infosys in Bengaluru invites an experienced Senior Data Engineer to lead end-to-end data pipelines using Azure Data Factory and Databricks. You will design ETL/ELT workflows, optimize PySpark transformations, and mentor the team while ensuring data quality and reliability.

You will collaborate with stakeholders to translate business needs into scalable data solutions and drive delivery planning, estimation, and risk management for data engineering initiatives.

Qualifications

• BTECH/MTECH/MCA/MSC in CS, Eng, or related field
• 7–9 years of data engineering with hands-on delivery ownership
• Expertise in Azure Data Factory for pipeline orchestration
• Databricks expertise for scalable data processing
• Hands-on PySpark for distributed transformations
• Production-grade pipelines with logging, error handling, and monitoring

Responsibilities

• Lead end-to-end data pipelines using ADF for orchestration and Databricks for processing
• Design robust ETL/ELT workflows with data quality
• Develop PySpark transformations for batch and incremental processing
• Build reusable frameworks, templates, and standards for pipeline development
• Define solution architecture for ingestion, transformation, and serving layers
• Tune Spark jobs for performance and cost efficiency
• Establish monitoring, alerting, and operational runbooks for production pipelines
• Provide technical leadership, code reviews, and mentoring
• Collaborate with stakeholders to translate business requirements into scalable data solutions
• Drive delivery planning, estimation, and risk management for data engineering initiatives

Skills

PySpark

Databricks

ADF orchestration

ETL/ELT workflows

Spark tuning

Azure

Data pipelines

Mentoring

Leadership

Observability

Education

BTech

MTech

MCA

MSc

Tools

Azure Data Factory

Databricks

PySpark

Spark

CI/CD

Job description

Technology->Big Data - Data Processing->PySpark Technology->Cloud Integration->Azure Data Factory (ADF) Technology->Data Engineering->Databricks

Key Responsibilities: Data Engineering & Delivery

• Lead end-to-end development of data pipelines using ADF for orchestration and Databricks for scalable processing
• Design and implement robust ETL/ELT workflows, ensuring data quality, reliability, and maintainability
• Develop optimized transformations and jobs using PySpark in Databricks for batch and incremental processing
• Build reusable frameworks, templates, and standards for pipeline development and deployment Architecture & Performance
• Define solution architecture for ingestion, transformation, and serving layers aligned to platform best practices
• Tune Spark jobs for performance and cost efficiency (partitioning, caching, shuffle optimization, file sizing)
• Establish monitoring, alerting, and operational runbooks for production pipelines Leadership & Collaboration
• Provide technical leadership, code reviews, and mentoring to ensure high engineering standards
• Collaborate with stakeholders to translate business requirements into scalable data solutions
• Drive delivery planning, estimation, and risk management for data engineering initiatives Minimum Qualifications:

Minimum Qualifications

• BTECH, MTECH, MCA, MSC (or equivalent) in Computer Science, Engineering, or related field
• 7–9 years of experience in data engineering with strong hands‑on delivery ownership
• Strong expertise in Azure Data Factory (ADF) for pipeline orchestration, scheduling, and integration patterns
• Strong expertise in Databricks for building scalable data processing solutions
• Hands‑on proficiency with PySpark for building and optimizing distributed data transformations
• Experience building production-grade pipelines with logging, error handling, and operational support readiness

Preferred Qualifications

• Experience designing medallion/layered data architectures and implementing reusable transformation patterns in Databricks
• Strong understanding of data modeling concepts and building curated datasets for analytics consumption
• Experience implementing CI/CD practices for data pipelines and notebooks, including automated testing and deployment
• Proven ability to lead technical discussions, mentor team members, and drive engineering best practices
• Experience improving observability (metrics, alerts, dashboards) and reducing pipeline failures through proactive monitoring