PySpark Data Engineer (AWS | Databricks | ETL)

Infosys; Hyderabad; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

Infosys is seeking a PySpark Engineer to design, develop, and maintain data pipelines using Azure Data Factory and Databricks. The role involves building scalable data processing solutions with PySpark and Python, and ensuring data quality across multiple sources.

You will optimize Spark jobs, implement ETL/ELT workflows, and collaborate with business stakeholders to meet data requirements while following CI/CD practices.

Qualifications

• Experience building scalable data pipelines in cloud environments.
• Strong knowledge of PySpark, Python, and Spark tuning for performance.
• Familiarity with CI/CD practices and version control in data projects.

Responsibilities

• Design, develop and maintain data pipelines using Azure Data Factory.
• Build data processing solutions with Azure Databricks, Python and PySpark.
• Optimize Spark jobs and notebooks for performance and scalability.

Skills

PySpark

Python

Data pipelines

Spark optimization

CI/CD

Tools

Azure Data Factory

Azure Databricks

Azure Data Lake Storage

Azure SQL

Job description

Job Description:

Position: PySpark Engineer

Roles & Responsibilities

• Design, develop, and maintain data pipelines using Azure Data Factory (ADF).
• Build scalable data processing solutions using Azure Databricks, Python, and PySpark.
• Develop ETL/ELT workflows for data ingestion, transformation, and loading.
• Work with Azure Data Lake Storage (ADLS), Azure SQL, and other Azure data services.
• Optimize Spark jobs and Databricks notebooks for performance and scalability.
• Implement data quality checks, validation rules, and monitoring processes.
• Create and manage end-to-end data integration solutions across multiple data sources.
• Collaborate with business stakeholders and data teams to understand data requirements.
• Troubleshoot and resolve data pipeline and performance issues.
• Follow coding standards, version control, and CI/CD best practices.
• Support data warehousing and analytics initiatives.
• Prepare technical documentation and maintain deployment artifacts.