Pyspark Developer (5 locations)

Tata Consultancy Services; Hyderabad, Chennai District, Bengaluru; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

Tata Consultancy Services is hiring for a PySpark/Data Engineer to design and optimize large-scale ETL/ELT pipelines across Bengaluru, Chennai, Hyderabad, Pune, and Kolkata. You will work with PySpark, Spark SQL, and Hadoop ecosystems to deliver reliable data processing solutions.

You will collaborate with data scientists and analysts, implement transformations, and ensure pipeline health with CI/CD practices using Git, Jenkins, or Azure DevOps.

Qualifications

• Strong Python and PySpark experience.
• Hands-on with Spark SQL, RDD, and DataFrame APIs.
• Good Hadoop knowledge: Hive/HDFS/YARN.
• Familiar with data formats: Parquet, Avro, JSON.
• Proficient in SQL and writing efficient queries.
• Experience with Airflow, Oozie, or similar tools.
• Version control using Git; CI/CD familiarity.

Responsibilities

• Design, develop, and maintain ETL/ELT pipelines via PySpark.
• Optimize Spark jobs for performance and scale.
• Transform, join, and aggregate large datasets; batch and real-time processing.
• Collaborate with data scientists and engineers; integrate with Hive/Redshift/Snowflake.
• Write clean, maintainable code; follow CI/CD with Git/Jenkins/Azure DevOps.
• Monitor data quality and pipeline health; ensure SLAs are met.
• Work with Hadoop, Hive, HDFS, AWS EMR, Databricks, or Azure Synapse as needed.

Skills

Python programming

SQL proficiency

Data processing

Collaboration

Troubleshooting

Tools

PySpark

Spark SQL

RDD

DataFrame APIs

Hadoop ecosystem

Hive

HDFS

YARN

Parquet

Avro

JSON

SQL

Airflow

Oozie

Git

CI/CD

Jenkins

Azure DevOps

AWS

GCP

Azure

Databricks

Azure Synapse

Redshift

Snowflake

Job description

Job Locations : Bengaluru, Chennai, Hyderabad, Pune, Kolkata

Job Requirements:

• Experience with strong proficiency in Python and PySpark.
• Experience working on Spark SQL, RDD, and DataFrame APIs.
• Good understanding of Hadoop ecosystem (Hive, HDFS, YARN).
• Knowledge of data formats like Parquet, Avro, JSON, etc.
• Experience with SQL and writing efficient queries.
• Familiarity with job orchestration tools (Airflow, Oozie, or similar).
• Version control systems like Git.
• Exposure to cloud platforms (AWS/GCP/Azure) is a plus.

Key responsibilities:

• Design, develop, and maintain robust ETL/ELT pipelines using PySpark and other big data technologies. Optimize Spark jobs for performance and scalability.
• Implement data transformations, aggregations, and joins over large datasets. Perform batch and real-time data processing tasks.
• Collaborate with data scientists, analysts, and other engineers to understand requirements and deliver quality solutions. Integrate PySpark solutions with data warehouses (like Hive, Redshift, Snowflake) and other data stores.
• Write clean, maintainable, and well-documented code. Follow version control and CI/CD practices using tools like Git, Jenkins, or Azure DevOps.
• Troubleshoot data quality and performance issues. Monitor pipeline health and ensure SLAs are met.
• Work with tools and platforms like Hadoop, Hive, HDFS, AWS EMR, Databricks, or Azure Synapse as required.