PySpark Developer | Python | SQL

Tata Consultancy Services; Bengaluru; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

Tata Consultancy Services is seeking a PySpark Developer to design and optimize large-scale data processing pipelines using Spark, Python, and SQL.

The role involves building batch and real-time ETL solutions, collaborating with data engineers and analysts, and ensuring quality and performance across Hadoop, Hive, and cloud platforms.

Qualifications

• Strong hands-on experience in Python and PySpark.
• Expertise in Spark SQL, DataFrames, and RDDs.
• Good knowledge of Hadoop (Hive, HDFS, YARN).
• Strong SQL and query optimization skills.
• Experience with Data Warehousing concepts.
• Knowledge of Parquet, Avro, JSON data formats.
• Experience with Git version control.
• Familiarity with Airflow, Oozie, or similar scheduling tools.
• Exposure to AWS, Azure, or GCP is an added advantage.

Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines using PySpark.

Skills

Python

PySpark

Spark SQL

DataFrames

RDDs

SQL

Data Warehousing

Parquet

Avro

JSON

Git

Airflow

Oozie

Cloud platforms

Tools

Hive

HDFS

YARN

AWS

Azure

GCP

CI/CD

Job description

Job Title:

PySpark Developer

Location:

Chennai / Bangalore / Hyderabad / Pune

Notice Period:

Immediate to 30 Days

Job Description :

We are seeking a skilled PySpark Developer with strong experience in Python, PySpark, SQL, and Data Warehousing concepts. The ideal candidate will be responsible for designing, developing, and optimizing large-scale data processing pipelines and ETL solutions using Spark-based technologies.

Key Responsibilities :

• Design, develop, and maintain scalable ETL/ELT pipelines using PySpark.
• Build and optimize Spark jobs for performance, reliability, and scalability.
• Process and transform large datasets using Spark SQL, DataFrames, and RDDs.
• Develop batch and real-time data processing solutions.
• Integrate data pipelines with Hive, HDFS, Snowflake, Redshift, and other data platforms.
• Collaborate with data engineers, analysts, and business stakeholders.
• Monitor data pipelines, troubleshoot issues, and ensure SLA compliance.
• Follow coding best practices, version control, and CI/CD processes.
• Work with Hadoop ecosystem tools and cloud platforms such as AWS, Azure, or GCP.

Required Skills:

• Strong hands-on experience in Python and PySpark
• Expertise in Spark SQL, DataFrames, and RDDs
• Good knowledge of Hadoop (Hive, HDFS, YARN)
• Strong SQL and query optimization skills
• Experience with Data Warehousing concepts
• Knowledge of Parquet, Avro, JSON data formats
• Experience with Git version control
• Familiarity with Airflow, Oozie, or similar scheduling tools
• Exposure to AWS, Azure, or GCP is an added advantage