ADF (Azure Data Factory)Databricks+Pyspark

Infosys; Bengaluru; On-site; INR 1,800,000 - 3,000,000; Full time

Job summary

Infosys in Bengaluru, India, seeks a Senior Data Engineer to lead the design and delivery of modern data pipelines using Azure Data Factory and Databricks. You will own end-to-end delivery, mentor engineers, and collaborate with product owners and analysts to shape scalable solutions.

Responsibilities span orchestration, batch and streaming processing with PySpark, plus building reusable templates, monitoring, and cost-efficient tuning.

Qualifications

• Bachelors or higher in CS/Engineering or related field.
• Strong hands-on data engineering delivery ownership.
• Expertise with ADF for orchestration and Databricks for processing.
• Proficient in PySpark for distributed data transformations.
• Experience building production-grade pipelines with logging and operations support.

Responsibilities

• Lead end-to-end development of data pipelines using ADF for orchestration and Databricks for scalable processing
• Design and implement robust ETL/ELT workflows, ensuring data quality, reliability, and maintainability
• Develop optimized transformations and jobs using PySpark in Databricks for batch and incremental processing
• Build reusable frameworks, templates, and standards for pipeline development and deployment
• Define solution architecture for ingestion, transformation, and serving layers aligned to platform best practices
• Tune Spark jobs for performance and cost efficiency (partitioning, caching, shuffle optimization, file sizing)
• Establish monitoring, alerting, and operational runbooks for production pipelines
• Provide technical leadership, code reviews, and mentoring to ensure high engineering standards
• Collaborate with stakeholders to translate business requirements into scalable data solutions
• Drive delivery planning, estimation, and risk management for data engineering initiatives

Skills

Azure Data Factory

Databricks

PySpark

Data Pipelines

Mentoring Engineers

Education

B.Tech / B.E. in CS/Engineering

M.Tech / MSc / MCA

Tools

Databricks Platform

Azure Data Factory (ADF)

PySpark

Job description

Job DescriptionYoull lead the design and delivery of modern data engineering solutions that turn raw data into trusted, analytics-ready assets. Working at the intersection of orchestration, scalable processing, and cloud-native platforms, youll partner closely with product owners, analysts, and engineering teams to build reliable pipelines that power business decisions. This role is ideal for someone who enjoys owning end-to-end deliveryshaping architecture, guiding implementation, and mentoring engineerswhile continuously improving performance, quality, and operational excellence. If youre excited by solving complex data challenges, enabling self-serve analytics, and building a collaborative culture that values craftsmanship and learning, this is the place to make a meaningful impact.

Roles Responsibilities

Key Responsibilities

• Lead end-to-end development of data pipelines using ADF for orchestration and Databricks for scalable processing
• Design and implement robust ETL/ELT workflows, ensuring data quality, reliability, and maintainability
• Develop optimized transformations and jobs using PySpark in Databricks for batch and incremental processing
• Build reusable frameworks, templates, and standards for pipeline development and deployment
• Define solution architecture for ingestion, transformation, and serving layers aligned to platform best practices
• Tune Spark jobs for performance and cost efficiency (partitioning, caching, shuffle optimization, file sizing)
• Establish monitoring, alerting, and operational runbooks for production pipelines
• Provide technical leadership, code reviews, and mentoring to ensure high engineering standards
• Collaborate with stakeholders to translate business requirements into scalable data solutions
• Drive delivery planning, estimation, and risk management for data engineering initiatives

Minimum Qualifications

• BTECH, MTECH, MCA, MSC (or equivalent) in Computer Science, Engineering, or related field
• 79 years of experience in data engineering with strong hands-on delivery ownership
• Strong expertise in Azure Data Factory (ADF) for pipeline orchestration, scheduling, and integration patterns
• Strong expertise in Databricks for building scalable data processing solutions
• Hands-on proficiency with PySpark for building and optimizing distributed data transformations
• Experience building production-grade pipelines with logging, error handling, and operational support readiness

Technical Requirement

• Technology->Big Data - Data Processing->PySpark
• Technology->Cloud Integration->Azure Data Factory (ADF)
• Technology->Data Engineering->Databricks

Preferred Qualifications

• Experience designing medallion/layered data architectures and implementing reusable transformation patterns in Databricks
• Strong understanding of data modeling concepts and building curated datasets for analytics consumption
• Experience implementing CI/CD practices for data pipelines and notebooks, including automated testing and deployment
• Proven ability to lead technical discussions, mentor team members, and drive engineering best practices
• Experience improving observability (metrics, alerts, dashboards) and reducing pipeline failures through proactive monitoring

Educational Requirement

Educational RequirementMCA,MSc,MTech,Bachelor of Engineering,BTech

Preferred Skills

Preferred SkillsTechnology->Cloud Integration->Azure Data Factory (ADF),Technology->Data Engineering->Databricks,Technology->Big Data - Data Processing->PySpark

Service Line

Service LineData Analytics Unit