Data Engineer | PySpark | Databricks | Azure Data Factory (Pan India)

Infosys; Hyderabad, Pune District, Bengaluru; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

Infosys in Hyderabad is seeking a data engineering professional to design, develop, and maintain data pipelines using Azure Data Factory, Databricks, Python, and PySpark. You will build scalable ETL/ELT workflows and collaborate with stakeholders to meet data requirements.

You will optimize Spark jobs, implement data quality checks, follow CI/CD practices, and document deployment artifacts to support analytics initiatives across multiple data sources.

Qualifications

• Experience in designing and building data pipelines on Azure stack.
• Strong understanding of ETL/ELT workflows and data quality.

Responsibilities

• Design, develop, and maintain data pipelines using Azure Data Factory (ADF).
• Build scalable data processing solutions using Azure Databricks, Python, and PySpark.
• Develop ETL/ELT workflows for data ingestion, transformation, and loading.
• Work with Azure Data Lake Storage (ADLS), Azure SQL, and other Azure data services.
• Optimize Spark jobs and Databricks notebooks for performance and scalability.
• Implement data quality checks, validation rules, and monitoring processes.
• Create and manage end-to-end data integration solutions across multiple data sources.
• Collaborate with business stakeholders and data teams to understand data requirements.
• Troubleshoot and resolve data pipeline and performance issues.
• Follow coding standards, version control, and CI/CD best practices.
• Support data warehousing and analytics initiatives.
• Prepare technical documentation and maintain deployment artifacts.

Skills

Data engineering

ETL/ELT

CI/CD

Collaboration with stakeholders

Tools

Azure Data Factory

Azure Databricks

Azure Data Lake Storage

Azure SQL

Python

PySpark

Job description

Roles & Responsibilities

• Design, develop, and maintain data pipelines using Azure Data Factory (ADF).
• Build scalable data processing solutions using Azure Databricks, Python, and PySpark.
• Develop ETL/ELT workflows for data ingestion, transformation, and loading.
• Work with Azure Data Lake Storage (ADLS), Azure SQL, and other Azure data services.
• Optimize Spark jobs and Databricks notebooks for performance and scalability.
• Implement data quality checks, validation rules, and monitoring processes.
• Create and manage end-to-end data integration solutions across multiple data sources.
• Collaborate with business stakeholders and data teams to understand data requirements.
• Troubleshoot and resolve data pipeline and performance issues.
• Follow coding standards, version control, and CI/CD best practices.
• Support data warehousing and analytics initiatives.
• Prepare technical documentation and maintain deployment artifacts.