Pyspark Developer

Infosys; Chennai District; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

Infosys in Chennai invites a senior data engineer to design and build large-scale data pipelines using PySpark and Hadoop in a global delivery environment.

You will work with Spark components, HDFS/Hive, and ETL concepts, participate in Agile ceremonies, and collaborate with stakeholders to deliver high-quality data solutions.

Qualifications

• 9+ years of experience designing large-scale data pipelines with PySpark and Hadoop.
• Strong problem solving and analytical skills.
• Experience in Global delivery environment.
• Excellent communication skills.
• At least 5 years in project development lifecycle activities.
• Experience with Data warehousing and ETL concepts.
• Unix shell scripting is a plus.
• Agile/Scrum environment experience.
• Ability to work with stakeholders and SMEs in eliciting requirements.

Responsibilities

• Translate business requirements into technical solutions.
• Collaborate with solution designers to scope and estimate projects.
• Elicit system requirements and apply domain knowledge.
• Participate in Agile ceremonies and adhere to sprint processes.
• Collaborate with multiple teams to troubleshoot and resolve data issues.

Skills

PySpark

Hadoop

Spark Core

Spark SQL

Batch processing

Spark Streaming

HDFS

Hive

Data warehousing

ETL

Unix shell scripting

Job description

Job Description

Primary skills: Pyspark, Hadoop, Spark Core, Spark SQL, Batch processing and Spark Streaming Hadoop, HDFS, Hive and other BigData technologies. Strong problem solving and Good Analytical skills.

• Excellent verbal and written communication skills.
• Experience and desire to work in a Global delivery environment.
• Stay up to date with new technologies and industry trends in Development.
• At least 9+ years of experience in designing and developing large scale, distributed data processing pipelines using PySpark, Hadoop and related technologies.
• Having expertise in Pyspark, Hadoop, Spark Core, Spark SQL, Batch processing and Spark Streaming
• Experience with Hadoop, HDFS, Hive and other BigData technologies.
• Familiarity with Data warehousing and ETL concepts and techniques
• UNIX shell scripting will be an added advantage in scheduling/running application jobs.
• At least 5 years of experience in Project development life cycle activities and development/maintenance projects
• Work with business stakeholders and other SMEs to understand high level business requirements.
• Work with the Solution Designers and contribute to the development of project plans by participating in the scoping and estimating of proposed project.
• Apply technical background understanding, business knowledge, system knowledge in the elicitation of Systems Requirements for projects.
• Possess good knowledge on Spark architecture and transformations using Spark and PySpark.
• Work in an Agile environment and participation in scrum daily standups, sprint planning reviews and retrospectives.
• Understand project requirements and translate them into technical solutions which meets the project quality standards
• Ability to work in team in diverse/multiple stakeholder environment and collaborate with upstream/downstream functional teams to identify, troubleshoot and resolve data issues.

Infosys is a global leader in next-generation digital services and consulting. Within the Data & Analytics unit, you will work on cutting-edge data engineering and analytics initiatives for global clients across industries. The role offers opportunities to work with cloud-native data platforms, modern analytics ecosystems, AI-driven solutions, and large-scale enterprise data modernization programs. Employees benefit from continuous learning programs, certifications, and career growth opportunities within Infosys' data and analytics practice

Requirements