Python-Pyspark Developer

Infosys; Hyderabad; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

Infosys in Hyderabad is seeking a Python PySpark Developer to design, develop, and optimize large-scale data processing systems. You will work on big data platforms, build scalable ETL pipelines, and process high-volume datasets using Spark and Python.

You will ingest data from databases, APIs, and files, integrate with Hadoop and cloud platforms, tune Spark jobs, and collaborate with data engineers and analysts to translate business requirements into technical solutions while ensuring

Qualifications

• Bachelor’s degree in engineering or related field.
• Experience with Python and PySpark preferred.
• Strong understanding of ETL processes.

Responsibilities

• Design, develop, and optimize large-scale data processing systems.
• Work on big data platforms, build scalable ETL pipelines, and process high-volume datasets using Spark and Python.
• Develop and maintain data pipelines using Python and PySpark.
• Process and transform large datasets in distributed environments.
• Build scalable ETL/ELT workflows.
• Work with Apache Spark (PySpark) for batch and real-time processing.
• Optimize Spark jobs for performance and efficiency.
• Handle structured and unstructured data.
• Ingest data from multiple sources: Databases (SQL/NoSQL), APIs, Files (CSV, JSON, Parquet).
• Integrate with data platforms like Hadoop (HDFS) and Cloud (AWS, Azure, GCP).
• Tune Spark jobs (partitioning, caching, parallelism).
• Optimize SQL queries and transformations.
• Improve data processing efficiency and cost.
• Collaborate with data engineers, data scientists, and analysts.
• Translate business requirements into technical solutions.
• Participate in code reviews and agile development practices.
• Debug and resolve issues in data pipelines.
• Monitor job execution and data quality.
• Ensure reliability and availability of data workflows.

Skills

Python

PySpark

Education

Bachelor of Engineering

BTech

BCA

BSc

MTech

MSc

MCA

Tools

Hadoop

AWS

Azure

GCP

Job description

Educational Requirements

• Bachelor of Engineering
• BTech
• BCA
• BSc
• MTech
• MSc
• MCA

Service Line

Data Analytics Unit

Responsibilities

• We are looking for an experienced Python PySpark Developer to design, develop, and optimize large-scale data processing systems.
• The ideal candidate will work on big data platforms, build scalable ETL pipelines, and process high-volume datasets using Spark and Python.
• Develop and maintain data pipelines using Python and PySpark.
• Process and transform large datasets in distributed environments.
• Build scalable ETL/ELT workflows.
• Work with Apache Spark (PySpark) for batch and real-time processing.
• Optimize Spark jobs for performance and efficiency.
• Handle structured and unstructured data.
• Ingest data from multiple sources: Databases (SQL/NoSQL), APIs, Files (CSV, JSON, Parquet).
• Integrate with data platforms like Hadoop (HDFS) and Cloud (AWS, Azure, GCP).
• Tune Spark jobs (partitioning, caching, parallelism).
• Optimize SQL queries and transformations.
• Improve data processing efficiency and cost.
• Collaborate with data engineers, data scientists, and analysts.
• Translate business requirements into technical solutions.
• Participate in code reviews and agile development practices.
• Debug and resolve issues in data pipelines.
• Monitor job execution and data quality.
• Ensure reliability and availability of data workflows.

Technical and Professional Requirements

• Primary skills: Python, PySpark

Preferred Skills

• OpenSystem
• Python
• PySpark
• Big Data
• Data Processing