PySpark Data Engineer

Infosys; Hyderabad, Pune District, Bengaluru; On-site; INR 900,000 - 1,500,000; Full time

Job summary

Infosys is seeking a PySpark Data Engineer to design and maintain scalable data pipelines using PySpark. The role emphasizes processing large datasets, ETL development, and collaboration with cross-functional teams to ensure data quality and operational excellence.

The ideal candidate will have 2-5 years of experience in data engineering, strong PySpark/Python/SQL skills, and knowledge of big data technologies.

Qualifications

• 2-5 years of experience in data engineering or related field.
• Strong hands-on experience with PySpark and Apache Spark.
• Proficiency in Python programming and SQL.
• Experience in ETL development and data pipelines.
• Understanding of big data concepts and distributed processing.

Responsibilities

• Design, develop, and maintain scalable data pipelines using PySpark.
• Process and transform large volumes of structured and unstructured data.
• Develop ETL workflows and data integration solutions.
• Optimize Spark jobs for performance and scalability.
• Collaborate with cross-functional teams to deliver data engineering solutions.
• Ensure data quality, reliability, and operational excellence.

Skills

PySpark

Apache Spark

Python

SQL

Data Engineering

ETL development

Big data concepts

Tools

Databricks

Hadoop

Hive

AWS

Azure

Job description

We are looking for a skilled PySpark Data Engineer with 2-5 years of experience in building scalable data processing solutions and data pipelines. The ideal candidate should have strong expertise in PySpark, Python, SQL, and big data technologies.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using PySpark.
• Process and transform large volumes of structured and unstructured data.
• Develop ETL workflows and data integration solutions.
• Optimize Spark jobs for performance and scalability.
• Collaborate with cross-functional teams to deliver data engineering solutions.
• Ensure data quality, reliability, and operational excellence.

Required Skills

• Strong hands-on experience with PySpark and Apache Spark.
• Proficiency in Python programming.
• Strong knowledge of SQL.
• Experience in Data Engineering and ETL development.
• Understanding of big data concepts and distributed data processing.

Good to Have

• Hadoop
• Hive
• Databricks
• AWS/Azure
• Data Warehousing