Pyspark Data Engineer

Tata Consultancy Services; Bengaluru; On-site; INR 2,500,000 - 4,200,000; Full time

Job summary

Tata Consultancy Services (TCS) in Bengaluru invites applications for a Pyspark Data Engineer. The ideal candidate has 6–10 years of experience and will design, develop, and maintain scalable data solutions on the AWS cloud.

You will build batch and real-time data pipelines, develop data warehousing with PySpark, convert Scala-based ETL and streaming jobs, optimize performance, and work with data architects, analysts, and scientists to deliver high-quality analytics platforms.

Qualifications

• 6–10 years of experience in AWS data engineering and data platforms.
• Strong experience building scalable data pipelines on AWS.
• Experience with PySpark and data warehousing.

Responsibilities

• Design, develop, and optimize scalable data pipelines using AWS services.
• Build batch and real-time data ingestion and processing frameworks.
• Develop enterprise-grade data warehousing solutions using PySpark.
• Convert Scala-based ETL, batch, and streaming pipelines into PySpark frameworks.
• Optimize PySpark jobs for performance, scalability, and reliability.
• Implement ETL/ELT processes for structured and unstructured data.
• Collaborate with Data Architects, Business Analysts, and Data Scientists to translate business requirements into technical solutions.

Skills

AWS

PySpark

ETL/ELT

Data Pipelines

Data Warehousing

CI/CD

IaC

Spark

Tools

Databricks

Apache Spark

Python

SQL

Job description

Greetings from Tata Consultancy Services (TCS)!!!

Job Title : Pyspark Data Engineer

Experience : 6- 10 Years

Job Requirements ;

We are seeking an experienced AWS Data Engineer to design, develop,

and maintain scalable data solutions on the Amazon Cloud Platform

(AWS). The ideal candidate will have strong expertise in building

modern data pipelines, data warehousing, big data processing, and

cloud-native analytics solutions. The role requires close collaboration

with business stakeholders, data architects, data scientists, and

application teams to deliver reliable and high-performance data

platforms.

Key Responsibilities

• Design, develop, and optimize scalable data pipelines using AWS services.
• Build and maintain batch and real-time data ingestion and processing frameworks.
• Develop enterprise-grade data warehousing solutions using PySpark.
• Convert Scala-based ETL, batch, and streaming pipelines into PySpark frameworks.
• Optimize PySpark jobs for performance, scalability, and
• Support cloud modernization initiatives on AWS/Databricks platforms.
• Implement ETL/ELT processes for structured and unstructured data.
• Integrate data from multiple sources including databases, APIs, files, and streaming platforms.
• Ensure data quality, governance, security, and compliance
• Automate deployment and operational processes using CI/CD and Infrastructure as Code (IaC).
• Monitor data pipelines and troubleshoot production issues.
• Collaborate with Data Architects, Business Analysts, and Data Scientists to translate business requirements into technical solutions.
• Implement data models, metadata management, and data
• Support migration of on-premises or multi-cloud data platforms to AWS.
• Lead the migration, modernization, and optimization of Apache Spark workloads on AWS EMR, including conversion of Scalabased Spark applications to PySpark, performance tuning, cluster optimization, dependency management, and ensuring scalable, cost-effective, and resilient data processing solutions.