Pyspark Developer

Infosys; Hyderabad; On-site; INR 1,500,000 - 2,400,000; Full time

Job summary

Infosys in Hyderabad is seeking a skilled PySpark Developer with 5-7 years of experience to build scalable data solutions on PySpark. The role requires strong expertise in PySpark, Spark, Hadoop, SQL and Hive exposure.

You will design and implement large-scale data pipelines, work with business stakeholders, participate in Agile ceremonies, and contribute to data warehousing and ETL initiatives. Familiarity with Unix shell scripting is a plus.

Qualifications

• 5+ years of experience designing and developing large-scale PySpark data pipelines.
• Expertise in PySpark, Hadoop, Spark Core/SQL, batch processing and Spark Streaming.
• Experience with Hadoop, HDFS, Hive and other Big Data technologies.
• Familiarity with data warehousing and ETL concepts.
• UNIX shell scripting for scheduling/running batch jobs (advantage).
• Strong understanding of SDLC and agile environments.

Responsibilities

• Design and develop large-scale distributed data pipelines using PySpark, Hadoop, and related tech.
• Expertise in Spark Core, Spark SQL, batch processing and Spark Streaming.
• Work with Hadoop, HDFS, Hive and other Big Data technologies.
• Contribute to data warehousing and ETL concepts.
• Leverage UNIX shell scripting when scheduling jobs.
• Collaborate with stakeholders and SMEs to understand business requirements.
• Participate with Solution Designers in scoping and estimating projects.
• Translate requirements into technical solutions meeting quality standards.
• Collaborate in an Agile environment with daily standups and sprints.

Skills

PySpark

Spark Core

Spark SQL

Spark Streaming

Hadoop

HDFS

Hive

Batch processing

UNIX shell scripting

Education

Master of Computer Applications

Master of Technology

Bachelor of Computer Applications

Bachelor of Science

Bachelor of Engineering

Bachelor of Technology

Tools

Sqoop

UNIX shell scripting

Job description

Job Description

We are seeking a skilled Pyspark Developer with 5-7 years of experience in building scalable solutions on Pyspark. The ideal candidate should have strong expertise in Pyspark with Spark, Hadoop, SQL exposure to Hive, Sqoop, UNIX shell scripting.

Roles Responsibilities

• At least 5+ years of experience in designing and developing large scale, distributed data processing pipelines using PySpark, Hadoop and related technologies.
  
• Having expertise in Pyspark, Hadoop, Spark Core, Spark SQL, Batch processing and Spark Streaming
  
• Experience with Hadoop, HDFS, Hive and other BigData technologies.
  
• Familiarity with Data warehousing and ETL concepts and techniques
  
• UNIX shell scripting will be an added advantage in scheduling/running application jobs.
  
• At least 49 years of experience in Project development life cycle activities and development/maintenance projects
  
• Work with business stakeholders and other SMEs to understand high level business requirements.
  
• Work with the Solution Designers and contribute to the development of project plans by participating in the scoping and estimating of proposed project.
  
• Apply technical background understanding, business knowledge, system knowledge in the elicitation of Systems Requirements for projects.
  
• Possess good knowledge on Spark architecture and transformations using Spark and PySpark.
  
• Work in an Agile environment and participation in scrum daily standups, sprint planning reviews and retrospectives.
  
• Understand project requirements and translate them into technical solutions which meets the project quality standards
  
• Ability to work in team in diverse/multiple stakeholder environment and collaborate with upstream/downstream functional teams to identify, troubleshoot and resolve data issues.
  

Technical Requirement

• Primary skills:
  Pyspark, Hadoop, Spark Core, Spark SQL, Batch processing and Spark Streaming
  
• Hadoop, HDFS, Hive and other BigData technologies.
  
• Strong problem solving and Good Analytical skills.
  

• Excellent verbal and written communication skills.
  
• Experience and desire to work in a Global delivery environment.
  
• Stay up to date with new technologies and industry trends in Development.
  

Additional Responsibility

Infosys is a global leader in next-generation digital services and consulting. Within the Data Analytics unit, you will work on cutting-edge data engineering and analytics initiatives for global clients across industries. The role offers opportunities to work with cloud-native data platforms, modern analytics ecosystems, AI-driven solutions, and large-scale enterprise data modernization programs. Employees benefit from continuous learning programs, certifications, and career growth opportunities within Infosys' data and analytics practice

Educational Requirement

Master Of Comp. Applications,Master Of Technology,Bachelor Of Comp. Applications,Bachelor Of Science,Bachelor of Engineering,Bachelor Of Technology

Preferred Skills

Technology->Big Data - Data Processing->PySpark

Service Line

Data Analytics Unit