Senior PySpark Developer

Infosys; Bengaluru; Presencial; INR 3.800.000 - 6.800.000; Jornada completa

Descripción de la vacante

Infosys in Bengaluru seeks a Senior PySpark Developer to design and implement scalable data pipelines using PySpark and Spark. You will optimize ETL/ELT frameworks for large-scale data and build high-performance Spark applications with Spark SQL, DataFrames, and RDDs.

Collaborate with data architects and business teams to ensure data quality and governance. The role requires deep SQL knowledge, experience with Hadoop ecosystem, data warehousing, and Unix/Linux proficiency, with a focus on

Formación

• 8+ years of IT experience.
• Strong hands-on experience in PySpark and Python.
• Expertise in Apache Spark, Spark SQL, DataFrames, and ETL Development.
• Strong SQL and database concepts.
• Experience with Hadoop Ecosystem (Hive, HDFS, YARN).
• Experience in Data Warehousing and Data Modeling.
• Hands-on experience with Unix/Linux.

Responsabilidades

• Design and develop scalable data pipelines using PySpark and Apache Spark.
• Build and optimize ETL/ELT frameworks for processing large volumes of structured and unstructured data.
• Develop high-performance Spark applications using Spark SQL, DataFrames, and RDDs.
• Work closely with data architects and business teams to define data requirements.
• Implement data quality, validation, and governance processes.
• Optimize Spark jobs for performance, memory management, and scalability.
• Integrate data from various sources including databases, APIs, cloud storage, and streaming platforms.
• Support production deployments and resolve performance bottlenecks.
• Mentor junior developers and participate in code reviews.

Conocimientos

PySpark

Python

Spark

Spark SQL

DataFrames

ETL

SQL concepts

Hive

Data Warehousing

Data Modeling

Unix/Linux

Herramientas

Hive

HDFS

YARN

Descripción del empleo

Senior PySpark Developer

Mandatory Skills

• 8+ years of IT experience.
• Strong hands-on experience in PySpark and Python.
• Expertise in Apache Spark, Spark SQL, DataFrames, and ETL Development.
• Strong SQL and database concepts.
• Experience with Hadoop Ecosystem (Hive, HDFS, YARN).
• Experience in Data Warehousing and Data Modeling.
• Hands-on experience with Unix/Linux.

Key Responsibilities

• Design and develop scalable data pipelines using PySpark and Apache Spark.
• Build and optimize ETL/ELT frameworks for processing large volumes of structured and unstructured data.
• Develop high-performance Spark applications using Spark SQL, DataFrames, and RDDs.
• Work closely with data architects and business teams to define data requirements.
• Implement data quality, validation, and governance processes.
• Optimize Spark jobs for performance, memory management, and scalability.
• Integrate data from various sources including databases, APIs, cloud storage, and streaming platforms.
• Support production deployments and resolve performance bottlenecks.
• Mentor junior developers and participate in code reviews.