AWS Pyspark Data Engineer

Tata Consultancy Services Limited; Indore District; On-site; INR 1,500,000 - 2,300,000; Full time

Job summary

Tata Consultancy Services Limited is seeking an experienced AWS PySpark Developer to design and implement scalable data pipelines on AWS. You will work with PySpark, Spark, and AWS data services to build data lakes, data warehouses, and analytics solutions.

You will develop batch and real-time processing, optimize Spark jobs for performance and cost, and collaborate with stakeholders across teams. 5+ years of experience and strong DevOps/CI/CD alignment are expected.

Qualifications

• Hands-on PySpark/Spark with large-scale data processing.
• Experience with AWS data services and data pipelines.
• Strong ETL/ELT development and data warehousing knowledge.

Responsibilities

• Design scalable ETL/ELT pipelines using PySpark.
• Build data ingestion frameworks on AWS.
• Develop batch and real-time data processing solutions.
• Optimize Spark jobs for performance and cost.
• Work with diverse datasets (structured, semi-structured, unstructured).
• Develop data lake/warehouse solutions and ensure data quality.
• Support deployments and troubleshoot production issues.
• Follow DevOps/CI/CD best practices.

Tools

Amazon S3

AWS Glue

AWS EMR

AWS Lambda

Amazon Redshift

AWS IAM

CloudWatch

AWS Step Functions

PySpark

Apache Spark

Spark SQL

DataFrames & RDDs

ETL/ELT Development

Data Modeling

Data Warehousing

Python

SQL

Shell Scripting

Redshift

PostgreSQL

MySQL

DynamoDB

Git

Jenkins/AWS CodePipeline

Docker

Terraform

Job description

Overall Experience - 5+

Location - Indore

Job Summary

We are seeking an experienced AWS PySpark Developer with strong expertise in building scalable data engineering solutions on AWS Cloud. The ideal candidate should have hands‑on experience in PySpark, Spark, AWS data services, ETL development, and large‑scale data processing. The role involves designing, developing, and optimizing cloud‑native data pipelines, data lakes, and analytics solutions.

Key Responsibilities

• Design and develop scalable ETL/ELT pipelines using PySpark.
• Build and maintain data ingestion frameworks on AWS.
• Develop batch and real‑time data processing solutions.
• Optimize Spark jobs for performance and cost efficiency.
• Work with structured, semi‑structured, and unstructured datasets.
• Develop data lake and data warehouse solutions.
• Implement data quality checks, monitoring, and alerting.
• Collaborate with business stakeholders, architects, and analytics teams.
• Support production deployments and troubleshoot critical issues.
• Follow DevOps and CI/CD best practices.

Mandatory Technical Skills

AWS Services

• Amazon S3
• AWS Glue
• AWS EMR
• AWS Lambda
• Amazon Redshift
• AWS IAM
• CloudWatch
• AWS Step Functions

Big Data & Data Engineering

• PySpark
• Apache Spark
• Spark SQL
• DataFrames & RDDs
• ETL/ELT Development
• Data Modeling
• Data Warehousing Concepts

Programming

• Python
• SQL
• Shell Scripting

Database Technologies

• Redshift
• PostgreSQL
• MySQL
• DynamoDB (Preferred)

DevOps & CI/CD

• Git
• Jenkins / AWS CodePipeline
• Docker
• Terraform (Preferred)