Pyspark Developer

Infosys; Maharashtra; On-site; INR 900,000 - 1,300,000; Full time

Job summary

Infosys is seeking a data engineering professional with 3+ years of experience in building large-scale distributed data processing pipelines using PySpark, Hadoop, and related technologies.

The role requires strong problem-solving and analytical skills, excellent communication, and experience in a Global delivery environment. Proficiency in Hadoop ecosystem and familiarity with ETL concepts is essential, with Unix shell scripting as an added advantage.

Qualifications

• 3+ years of experience designing and developing large-scale distributed data processing pipelines with PySpark and Hadoop.
• Proficient with PySpark, Hadoop, Spark Core, Spark SQL, Batch and Spark Streaming.
• Experience with Hadoop, HDFS, Hive and other BigData technologies.
• Familiar with data warehousing and ETL concepts.
• Unix shell scripting is a plus for scheduling/running jobs.
• Experience in project development lifecycle activities and maintenance projects.
• Work with stakeholders to understand requirements; participate in scoping and estimating.
• Knowledge of Spark architecture and transformations using Spark and PySpark.
• Work in an Agile environment and participate in Scrum ceremonies.

Responsibilities

• Design and develop large-scale, distributed data processing pipelines using PySpark and Hadoop.
• Collaborate with business stakeholders and SMEs to understand high-level requirements.
• contribute to project plans by scoping and estimating proposed work.
• Translate requirements into technical solutions meeting project quality standards.
• Work with upstream/downstream teams to troubleshoot and resolve data issues.
• Participate in daily standups, sprint planning, reviews, and retrospectives.

Skills

Pyspark

Hadoop

Spark Core

Spark SQL

Spark Streaming

HDFS

Hive

Data warehousing

ETL concepts

UNIX shell scripting

Agile/Scrum

Spark architecture

Job description

Primary skills: Pyspark, Hadoop, Spark Core, Spark SQL, Batch processing and Spark Streaming Hadoop, HDFS, Hive and other BigData technologies. Strong problem solving and Good Analytical skills.

• Excellent verbal and written communication skills.
• Experience and desire to work in a Global delivery environment.
• Stay up to date with new technologies and industry trends in Development.
• At least 3+ years of experience in designing and developing large scale, distributed data processing pipelines using PySpark, Hadoop and related technologies.
• Having expertise in Pyspark, Hadoop, Spark Core, Spark SQL, Batch processing and Spark Streaming
• Experience with Hadoop, HDFS, Hive and other BigData technologies.
• Familiarity with Data warehousing and ETL concepts and techniques
• UNIX shell scripting will be an added advantage in scheduling/running application jobs.
• At least 3 years of experience in Project development life cycle activities and development/maintenance projects
• Work with business stakeholders and other SMEs to understand high level business requirements.
• Work with the Solution Designers and contribute to the development of project plans by participating in the scoping and estimating of proposed project.
• Apply technical background understanding, business knowledge, system knowledge in the elicitation of Systems Requirements for projects.
• Possess good knowledge on Spark architecture and transformations using Spark and PySpark.
• Work in an Agile environment and participation in scrum daily standups, sprint planning reviews and retrospectives.
• Understand project requirements and translate them into technical solutions which meets the project quality standards
• Ability to work in team in diverse/multiple stakeholder environment and collaborate with upstream/downstream functional teams to identify, troubleshoot and resolve data issues.

Infosys is a global leader in next-generation digital services and consulting. Within the Data & Analytics unit, you will work on cutting-edge data engineering and analytics initiatives for global clients across industries. The role offers opportunities to work with cloud-native data platforms, modern analytics ecosystems, AI-driven solutions, and large-scale enterprise data modernization programs. Employees benefit from continuous learning programs, certifications, and career growth opportunities within Infosys' data and analytics practice