AWS Spark Data Engineer

Tata Consultancy Services; Chennai District; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

Tata Consultancy Services is seeking a data engineer to design, develop and scale end-to-end data pipelines using Spark (Scala) on AWS. You will build ETL/ELT workflows, work with large datasets, and implement governance and security best practices across services like S3, EMR, Glue, Redshift and Athena.

The role requires strong SQL, distributed computing knowledge, and hands-on experience with Airflow or Step Functions.

Qualifications

• Experience in designing scalable data pipelines using Spark and Scala.
• Hands-on expertise with AWS data services and cloud infrastructure.
• Strong SQL skills and data governance understanding.
• Familiarity with distributed computing concepts and best practices.
• Experience with data orchestration tools such as Airflow or Step Functions.

Responsibilities

• Design, develop and maintain scalable data pipelines using Apache Spark (Scala).
• Build and optimize ETL/ELT workflows on AWS.
• Work with large-scale structured and unstructured datasets.
• Develop data solutions using AWS services such as S3, EMR, Glue, Redshift, Athena, Lambda.
• Ensure data security, governance and compliance best practices.

Skills

AWS Data Engineering

Spark

Scala

Distributed computing

SQL

Tools

Airflow

Step Functions

S3

EMR

Glue

Redshift

Athena

Lambda

Job description

Desired Competencies (Technical/Behavioral Competency)

• Strong Expertise in AWS Data Engineering
• Hands-on expertise in Spark and Scala
• Extensive experience with AWS Cloud Services (S3, EMR, Glue, Redshift, Athena etc)
• Solid understanding of Distributed computing
• Proficiency in SQL
• Experience in Data Pipeline Orchestration tools (Airflow, Step Functions etc)

Responsibility of / Expectations from the Role

Design Develop and maintain Scalable Data Pipelines using Apache Spark (Scala)

Build and Optimize ETL/ELT workflows on AWS

Work with large scale structured and unstructured Datasets

Develop Data Solutions using AWS services such as S3, EMR, Glue, Redshift, Athena,Lambda

Ensure Data Security, Governance and Compliance Best Practices