AWS Pyspark Developer

Tata Consultancy Services Limited; Indore District; On-site; INR 1,200,000 - 2,400,000; Full time

Job summary

Tata Consultancy Services Limited in Indore, India, is seeking an experienced AWS PySpark Developer to design and build scalable data pipelines on the AWS Cloud. You will work with PySpark, Spark, and AWS data services to deliver cloud-native ETL/ELT solutions and data lakes.

The role involves developing batch and real-time data processing, optimizing Spark jobs for performance and cost, and collaborating with analytics teams and stakeholders.

Qualifications

• Hands-on experience in PySpark, Spark, AWS data services, ETL development, and large-scale data processing.
• Experience with designing and implementing cloud-native data pipelines, data lakes and data warehouses on AWS.

Responsibilities

• Design and develop scalable ETL/ELT pipelines using PySpark.
• Build and maintain data ingestion frameworks on AWS.
• Develop batch and real‑time data processing solutions.
• Optimize Spark jobs for performance and cost efficiency.
• Work with structured, semi‑structured, and unstructured datasets.
• Develop data lake and data warehouse solutions.
• Implement data quality checks, monitoring, and alerting.
• Collaborate with business stakeholders, architects, and analytics teams.
• Support production deployments and troubleshoot critical issues.
• Follow DevOps and CI/CD best practices.

Skills

PySpark

Spark

ETL/ELT

Data pipelines

Data lakes

Data modeling

Tools

Amazon S3

AWS Glue

AWS EMR

AWS Lambda

Redshift

AWS IAM

CloudWatch

Step Functions

Git

Jenkins

CodePipeline

Docker

Terraform

Job description

Job Summary

We are seeking an experienced AWS PySpark Developer with strong expertise in building scalable data engineering solutions on AWS Cloud. The ideal candidate should have hands‑on experience in PySpark, Spark, AWS data services, ETL development, and large‑scale data processing. The role involves designing, developing, and optimizing cloud‑native data pipelines, data lakes, and analytics solutions.

Key Responsibilities

• Design and develop scalable ETL/ELT pipelines using PySpark.
• Build and maintain data ingestion frameworks on AWS.
• Develop batch and real‑time data processing solutions.
• Optimize Spark jobs for performance and cost efficiency.
• Work with structured, semi‑structured, and unstructured datasets.
• Develop data lake and data warehouse solutions.
• Implement data quality checks, monitoring, and alerting.
• Collaborate with business stakeholders, architects, and analytics teams.
• Support production deployments and troubleshoot critical issues.
• Follow DevOps and CI/CD best practices.

Mandatory Technical Skills

AWS Services

• Amazon S3
• AWS Glue
• AWS EMR
• AWS Lambda
• Amazon Redshift
• AWS IAM
• CloudWatch
• AWS Step Functions

Big Data & Data Engineering

• PySpark
• Apache Spark
• Spark SQL
• DataFrames & RDDs
• ETL/ELT Development
• Data Modeling
• Data Warehousing Concepts

Programming

• Python
• SQL
• Shell Scripting

Database Technologies

• Redshift
• PostgreSQL
• MySQL
• DynamoDB (Preferred)

DevOps & CI/CD

• Git
• Jenkins / AWS CodePipeline
• Docker
• Terraform (Preferred)

Good-to-Have Skills

• Databricks
• Kafka
• Apache Airflow
• Delta Lake
• Snowflake
• Kubernetes
• Streaming Technologies
• Machine Learning Exposure