Pyspark Developer

Tata Consultancy Services; Kolkata District, Hyderabad, Chennai District; On-site; INR 2,600,000 - 3,800,000; Full time

Job summary

Tata Consultancy Services invites a PySpark Data Engineer to join our data platforms in India. The role focuses on designing, developing, and maintaining robust data pipelines using PySpark/Python to support enterprise data processing.

You will migrate Ab Initio ETL workloads to PySpark, work with on-prem and cloud platforms (AWS/Databricks/Snowflake), and ensure data quality, reliability, and performance across large-scale datasets while collaborating with cross-functional teams.

Qualifications

• 5+ years of experience in data engineering or related role.
• Design, develop and maintain scalable data pipelines with PySpark and Python.
• Experience migrating ETL workloads to PySpark and cloud platforms.

Responsibilities

• Design, develop, and maintain robust, scalable data pipelines using PySpark.
• Create and manage ETL processes for data migration and deployment across systems.
• Migrate Ab Initio ETL applications into PySpark-based data pipelines.
• Migrate on-premise workloads to cloud platforms such as AWS, Databricks, and Snowflake.
• Collaborate with cross-functional teams to resolve data-related issues.
• Ensure data quality, reliability, and performance of data solutions.
• Stay updated with emerging data engineering technologies and best practices.

Skills

PySpark

Python

Hadoop

AWS

Databricks

Snowflake

Airflow

CI/CD

Git

Debugging

Communication

Tools

Databricks

Snowflake

Airflow

Docker

Job description

Tata Consultancy Services (TCS) is Hiring!

PySpark Data Engineer

Locations: Chennai, Kolkata, Hyderabad, Pune

Experience: 5+ Years

Job Description

As a Data Engineer, you will be responsible for designing, developing, and maintaining data solutions for data generation, collection, and processing in Big Data environments using predominantly PySpark/Python. The role involves building scalable data pipelines, ensuring data quality, and implementing ETL processes to support enterprise data platforms.

Key Responsibilities

• Design, develop, and maintain robust, scalable, high-performance data pipelines using PySpark.


• Create and manage ETL processes for data migration and deployment across systems.


• Migrate Ab Initio ETL applications into PySpark-based data pipelines.


• Migrate on-premise workloads to cloud platforms such as AWS, Databricks, and Snowflake.


• Collaborate with cross-functional teams to resolve data-related issues.


• Ensure data quality, reliability, and performance of data solutions.


• Stay updated with emerging data engineering technologies and best practices.

Required Skills

• Strong experience in PySpark and Python


• Hands-on experience with Hadoop Ecosystem


• Experience handling large-scale data processing and transformation


• AWS Cloud services


• Databricks and/or Snowflake


• Apache Airflow


• CI/CD pipelines and Git


• Strong debugging and problem-solving skills


• Excellent communication and collaboration skills

Good to Have

• AWS EKS


• Docker & Containers