PySpark / Spark Developer

Infosys; Hyderabad, Pune District, Bengaluru; On-site; INR 1,500,000 - 2,300,000; Full time

Job summary

Infosys in Hyderabad is seeking an experienced PySpark/Spark Developer to design, develop, and optimize large-scale data processing solutions. You will build ETL pipelines, implement Spark SQL workflows, and ensure data quality across distributed platforms.

The role requires 5-8 years in data engineering with strong Python/SQL skills, plus familiarity with Spark, Hive, and cloud tech. Collaboration with business and architecture teams is essential.

Qualifications

• 5-8 years of experience in Data Engineering and Big Data technologies.
• Strong hands-on experience in PySpark and Apache Spark.
• Excellent Python and SQL skills.
• Experience with Spark SQL, ETL development, and Data Warehousing.
• Experience working with large-scale distributed data platforms.
• Exposure to Databricks, Hadoop, Hive, or Spark Streaming is preferred.
• Knowledge of AWS or Azure cloud platforms is an added advantage.
• Strong analytical and problem-solving skills.
• Good communication and stakeholder management abilities.

Responsibilities

• Design, develop, and maintain scalable data processing solutions using PySpark and Apache Spark.
• Build and optimize ETL pipelines for processing large datasets.
• Develop data transformation workflows using Spark SQL and Python.
• Work with Hadoop ecosystem technologies including Hive and HDFS.
• Optimize Spark jobs for performance, scalability, and reliability.
• Develop and support batch and large-scale distributed processing applications.
• Collaborate with business stakeholders and technical teams to understand requirements.
• Troubleshoot and resolve data processing and performance issues.
• Ensure data quality, governance, and operational excellence.
• Participate in Agile ceremonies and project delivery activities.

Skills

Python

SQL

Analytical thinking

Stakeholder management

Tools

Apache Spark

Spark SQL

Hive

HDFS

Databricks

Hadoop

Spark Streaming

AWS

Azure

Job description

We are looking for an experienced PySpark/Spark Developer with 5-8 years of experience in designing, developing, and optimizing big data solutions. The ideal candidate should possess strong expertise in PySpark, Apache Spark, SQL, and distributed data processing technologies.

The candidate will work closely with business stakeholders, architects, and development teams to build scalable and high-performance data engineering solutions.

Role & Responsibilities

• Design, develop, and maintain scalable data processing solutions using PySpark and Apache Spark.
• Build and optimize ETL pipelines for processing large datasets.
• Develop data transformation workflows using Spark SQL and Python.
• Work with Hadoop ecosystem technologies including Hive and HDFS.
• Optimize Spark jobs for performance, scalability, and reliability.
• Develop and support batch and large-scale distributed processing applications.
• Collaborate with business stakeholders and technical teams to understand requirements.
• Troubleshoot and resolve data processing and performance issues.
• Ensure data quality, governance, and operational excellence.
• Participate in Agile ceremonies and project delivery activities.

Preferred Candidate Profile

• 5-8 years of experience in Data Engineering and Big Data technologies.
• Strong hands-on experience in PySpark and Apache Spark.
• Excellent Python and SQL skills.
• Experience with Spark SQL, ETL development, and Data Warehousing.
• Experience working with large-scale distributed data platforms.
• Exposure to Databricks, Hadoop, Hive, or Spark Streaming is preferred.
• Knowledge of AWS or Azure cloud platforms is an added advantage.
• Strong analytical and problem-solving skills.
• Good communication and stakeholder management abilities.