Data Engineer - PySpark | Spark | SQL

Infosys; Bengaluru; On-site; INR 1,200,000 - 2,500,000; Full time

Job summary

Infosys in Bengaluru, India seeks a PySpark and Databricks Engineer to design, build, and optimize data pipelines and ETL processes within the Azure ecosystem.

You will work with ADF, Databricks, PySpark, and Python to deliver scalable data solutions, enforce data quality, and collaborate with stakeholders across analytics teams.

Qualifications

• Experience designing and maintaining data pipelines with Azure Data Factory.
• Experience with Azure Databricks, PySpark, and Python for data processing.
• Experience developing ETL/ELT workflows and data ingestion.
• Familiarity with ADLS, Azure SQL, and other Azure data services.
• Knowledge of data quality checks and monitoring.

Responsibilities

• Design, develop, and maintain data pipelines using Azure Data Factory.
• Build scalable data processing solutions using Azure Databricks, Python, and PySpark.
• Develop ETL/ELT workflows for data ingestion, transformation, and loading.
• Work with Azure Data Lake Storage (ADLS), Azure SQL, and other Azure data services.
• Optimize Spark jobs and Databricks notebooks for performance and scalability.
• Implement data quality checks, validation rules, and monitoring processes.
• Create and manage end-to-end data integration solutions across multiple data sources.
• Collaborate with business stakeholders and data teams to understand data requirements.
• Troubleshoot and resolve data pipeline and performance issues.
• Follow coding standards, version control, and CI/CD best practices.
• Support data warehousing and analytics initiatives.
• Prepare technical documentation and maintain deployment artifacts.

Skills

Python

ETL/ELT

SQL

CI/CD

Tools

Azure Data Factory

Azure Databricks

Azure Data Lake Storage

Spark

Azure SQL

Job description

Job Description:

Position: PySpark , Databricks Engineer

Roles & Responsibilities

• Design, develop, and maintain data pipelines using Azure Data Factory (ADF).
• Build scalable data processing solutions using Azure Databricks, Python, and PySpark.
• Develop ETL/ELT workflows for data ingestion, transformation, and loading.
• Work with Azure Data Lake Storage (ADLS), Azure SQL, and other Azure data services.
• Optimize Spark jobs and Databricks notebooks for performance and scalability.
• Implement data quality checks, validation rules, and monitoring processes.
• Create and manage end-to-end data integration solutions across multiple data sources.
• Collaborate with business stakeholders and data teams to understand data requirements.
• Troubleshoot and resolve data pipeline and performance issues.
• Follow coding standards, version control, and CI/CD best practices.
• Support data warehousing and analytics initiatives.
• Prepare technical documentation and maintain deployment artifacts.