Data Engineer

Tata Consultancy Services Limited; Indore District; On-site; INR 1,800,000 - 4,000,000; Full time

Job summary

Tata Consultancy Services Limited is seeking an experienced AWS PySpark Data Engineer in Indore, India. The role focuses on building scalable ETL/ELT pipelines, processing large-scale datasets with PySpark, and delivering cloud-native data solutions on AWS.

Key responsibilities include designing and maintaining data pipelines, optimizing Spark workloads, ensuring data quality and governance, and collaborating with data architects and stakeholders in an Agile setup.

Qualifications

• 5+ years of overall experience in data engineering or related field.
• Proficient in Python and SQL with strong problem-solving skills.
• Hands-on experience with AWS Glue and S3, EMR, Lambda, Redshift.
• Knowledge of Data Lake/Data Warehouse concepts and ETL/ELT design.
• Experience with large-scale distributed data processing and Spark/PySpark.

Responsibilities

• Design, develop and maintain scalable ETL/ELT pipelines using PySpark.
• Build data ingestion and transformation frameworks on AWS.
• Process large volumes of structured, semi-structured and unstructured data.
• Develop and optimize Spark/PySpark workloads.
• Build cloud-native Data Lake and Data Warehouse solutions.
• Implement data quality, validation, monitoring and governance.
• Monitor and troubleshoot production data pipelines.
• Participate in CI/CD, deployments and Agile delivery.
• Collaborate with data architects, analytics teams and stakeholders.

Skills

Python

SQL

AWS Glue

Amazon S3

AWS EMR

AWS Lambda

Amazon Redshift

Data lake concepts

Distributed data processing

Job description

Job Description: AWS PySpark Data Engineer

Job Title: AWS PySpark Data Engineer

Experience: 5+ Years

Job Summary

We are looking for an experienced AWS PySpark Data Engineer with strong hands-on expertise in building scalable data engineering solutions on AWS. The role will focus on developing and optimizing ETL/ELT pipelines, processing large-scale datasets using PySpark/Spark, and building cloud-native data solutions using AWS services.

Mandatory Skills

• 5+ years of overall experience
• Python
• SQL
• AWS Glue
• Amazon S3
• AWS EMR
• AWS Lambda
• Amazon Redshift
• Data Lake/Data Warehouse concepts
• Large-scale/distributed data processing

Key Responsibilities

• Design, develop and maintain scalable ETL/ELT pipelines using PySpark.
• Build data ingestion and transformation frameworks on AWS.
• Process large volumes of structured, semi-structured and unstructured data.
• Develop and optimize Spark/PySpark workloads.
• Build cloud-native Data Lake and Data Warehouse solutions.
• Implement data quality, validation, monitoring and governance.
• Monitor and troubleshoot production data pipelines.
• Participate in CI/CD, deployments and Agile delivery.
• Work closely with data architects, analytics teams and business stakeholders.

Core: S3, Glue, EMR, Lambda, Redshift

Additional: Athena, IAM, CloudWatch, Step Functions