PySpark ETL Developer (AWS Glue | Spark | SQL)

Infosys; Pune District; On-site; INR 600,000 - 900,000; Full time

Job summary

Infosys is seeking a PySpark Engineer in Pune to design and maintain robust data pipelines using Azure Data Factory, Databricks, and PySpark. You will develop ETL/ELT workflows, work with ADLS and Azure SQL, and optimize Spark jobs for performance.

You will collaborate with data teams and stakeholders to understand requirements, implement quality checks, and ensure deployment artifacts are prepared and documented for smooth CI/CD processes.

Qualifications

• Experience building data pipelines with PySpark and Databricks
• Strong knowledge of ETL/ELT processes and data integration
• Familiarity with Azure data services and storage systems
• Ability to optimize Spark jobs for performance and scalability
• Experience with CI/CD practices for data projects

Responsibilities

• Design, develop, and maintain data pipelines using Azure Data Factory
• Build scalable data processing solutions with Azure Databricks and PySpark
• Develop ETL/ELT workflows for ingestion, transformation, and loading
• Work with ADLS and Azure SQL and other data services
• Optimize Spark jobs and notebooks for performance and scalability
• Implement data quality checks and monitoring
• Create end-to-end data integration across multiple sources
• Collaborate with stakeholders to understand data requirements
• Troubleshoot data pipeline issues and enforce coding standards
• Prepare deployment artifacts and technical documentation

Skills

PySpark

Azure Databricks

Python

ADF

Data Pipelines

ETL/ELT

ADLS/ADLS

CI/CD

Tools

Azure Data Factory

Azure SQL

Databricks notebooks

Job description

Job Description:

Position: PySpark Engineer

Roles & Responsibilities

• Design, develop, and maintain data pipelines using Azure Data Factory (ADF).
• Build scalable data processing solutions using Azure Databricks, Python, and PySpark.
• Develop ETL/ELT workflows for data ingestion, transformation, and loading.
• Work with Azure Data Lake Storage (ADLS), Azure SQL, and other Azure data services.
• Optimize Spark jobs and Databricks notebooks for performance and scalability.
• Implement data quality checks, validation rules, and monitoring processes.
• Create and manage end-to-end data integration solutions across multiple data sources.
• Collaborate with business stakeholders and data teams to understand data requirements.
• Troubleshoot and resolve data pipeline and performance issues.
• Follow coding standards, version control, and CI/CD best practices.
• Support data warehousing and analytics initiatives.
• Prepare technical documentation and maintain deployment artifacts.