Data Engineer (Spark & Python)

Infosys; Hyderabad, Pune District, Bengaluru; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

Infosys is seeking a Spark Developer to design and implement scalable data processing solutions using PySpark and Spark.

The role involves building ETL/ELT pipelines, processing structured and unstructured data, and optimizing Spark jobs for performance and reliability across cloud and on-premises environments.

Qualifications

• Must design and develop scalable data processing apps using PySpark and Spark.
• Experience building ETL/ELT pipelines for large datasets.
• Proficient in Spark SQL and DataFrames; able to optimize Spark jobs.

Responsibilities

• Design and develop scalable data processing applications using PySpark and Spark.
• Build and maintain ETL/ELT pipelines for large-scale datasets.
• Process structured and unstructured data from multiple sources.
• Optimize Spark jobs for performance, scalability, and reliability.
• Develop data transformations using Spark SQL and DataFrames.
• Integrate data from cloud and on-premises systems.
• Work with Hadoop, Hive, Databricks, and Data Lake architectures.
• Troubleshoot performance issues in Spark applications.
• Implement data quality checks and validation processes.
• Collaborate with data engineers, analysts, and business stakeholders.
• Support batch and real-time data processing solutions.
• Maintain technical documentation and coding standards.

Skills

PySpark

Spark

ETL/ELT

Spark SQL

DataFrames

Hadoop

Hive

Databricks

Data Lake

Real-time processing

Tools

Databricks

Hadoop

Hive

Job description

Position: Spark Developer

Experience: 5 to 8 years

Location: Pan India Opening

Employment Type: Full-Time

Roles & Responsibilities

• Design and develop scalable data processing applications using PySpark and Spark.
• Build and maintain ETL/ELT pipelines for large-scale datasets.
• Process structured and unstructured data from multiple sources.
• Optimize Spark jobs for performance, scalability, and reliability.
• Develop data transformations using Spark SQL and DataFrames.
• Integrate data from cloud and on-premises systems.
• Work with Hadoop, Hive, Databricks, and Data Lake architectures.
• Troubleshoot and resolve performance issues in Spark applications.
• Implement data quality checks and validation processes.
• Collaborate with data engineers, analysts, and business stakeholders.
• Support batch and real-time data processing solutions.
• Maintain technical documentation and coding standards.