Senior AWS Data Engineer (Spark / EMR / Glue / Iceberg)

Tata Consultancy Services; Hyderabad, Bengaluru; On-site; INR 2,500,000 - 4,000,000; Full time

Job summary

Tata Consultancy Services in Hyderabad is seeking a Senior AWS Data Engineer to design, build and optimize scalable data ingestion, transformation and analytics pipelines across enterprise Lakehouse architectures on AWS.

You will work with Spark, Iceberg, AWS EMR/Glue, S3 and streaming systems, ensuring data governance, data quality and performance while collaborating with data architects, analytics teams and platform engineers in an Agile environment.

Qualifications

• 6–10 years of experience in data engineering.
• Strong expertise in Spark, PySpark, AWS EMR, Glue, Iceberg.
• Experience building Data Lakehouses and data pipelines.
• Hands-on with batch and streaming processing.
• SQL and data modeling skills; experience with AWS cloud services.
• Understanding of data governance and data lineage concepts.

Responsibilities

• Design and develop scalable data ingestion and transformation pipelines.
• Build Data Lakehouse architectures using Apache Iceberg.
• Implement high-performance data processing using Apache Spark.
• Develop reusable data engineering frameworks.
• Design data solutions on AWS EMR, Glue, S3, Lambda.
• Collaborate with architects and analytics teams; participate in Agile delivery.

Skills

Apache Spark

PySpark

AWS EMR

AWS Glue

Apache Iceberg

Data Lakehouse

Data Pipelines

SQL

Data Governance

Performance Tuning

Tools

Airflow

Delta Lake

Apache Kafka

AWS Kinesis

Job description

We are looking for an experienced Senior AWS Data Engineer with strong expertise in Apache Spark, AWS EMR, AWS Glue, Apache Iceberg, Data Lakehouse Architecture, and AWS Cloud Data Services. The ideal candidate will be responsible for designing, developing, and optimizing scalable data ingestion, transformation, and analytics pipelines while supporting enterprise-wide data engineering and modernization initiatives.

The role requires hands-on experience in building cloud-native data platforms, batch and streaming architectures, data governance frameworks, and high-performance data lakehouse solutions.

Key Responsibilities

Data Engineering & Lakehouse Development

• Design and develop scalable data ingestion and transformation pipelines.
• Build enterprise-grade Data Lakehouse architectures using Apache Iceberg.
• Implement high-performance data processing solutions using Apache Spark.
• Develop reusable and scalable data engineering frameworks.

AWS Data Platform Engineering

• Design and implement data solutions using:
  • AWS EMR
  • AWS Glue
  • Amazon S3
  • AWS Lambda
  • AWS Analytics Services
• Build secure, scalable, and reliable data platforms on AWS.
• Optimize storage, processing, and data access strategies.

Batch, Streaming & Event-Driven Architectures

• Develop batch and real-time data processing pipelines.
• Implement:
  • Streaming Architectures
  • Event-Driven Architectures
  • CDC (Change Data Capture) Solutions
• Enable near real-time analytics and data delivery.

Data Quality & Governance

• Implement data quality monitoring and validation frameworks.
• Establish data lineage and governance processes.
• Ensure compliance with enterprise data standards and best practices.
• Maintain metadata, cataloging, and auditing capabilities.

Performance Optimization

• Optimize Spark jobs and AWS workloads for performance and cost efficiency.
• Tune data pipelines, storage structures, and processing frameworks.
• Improve throughput, scalability, and reliability of data platforms.

Collaboration & Delivery

• Work closely with data architects, business teams, analytics teams, and platform engineers.
• Participate in Agile delivery processes and sprint activities.
• Contribute to code reviews and engineering best practices.
• Support deployment, testing, and production operations.

Required Skills

• 6-10 years of experience in Data Engineering.
• Strong expertise in:
  • Apache Spark
  • PySpark
  • AWS EMR
  • AWS Glue
  • Apache Iceberg
• Experience building:
  • Data Lakehouses
  • Data Pipelines
  • Data Integration Solutions
• Hands-on experience with:
  • Batch Processing
  • Streaming Pipelines
  • CDC Frameworks
  • Event-Driven Architectures
• Strong SQL and Data Modeling skills.
• Experience working with AWS Cloud services.
• Understanding of Data Governance and Data Lineage concepts.
• Experience with performance tuning and optimization.

Preferred Skills

• Apache Kafka.
• AWS Kinesis.
• Delta Lake.
• Airflow.
• Data Catalog and Metadata Management frameworks.
• Data Quality Frameworks.
• CI/CD for Data Engineering.
• Banking or Financial Services experience.

Desired Candidate Profile

• Strong analytical and problem-solving skills.
• Excellent communication and stakeholder management abilities.
• Experience working with enterprise-scale data platforms.
• Ability to work independently and collaboratively in Agile teams.
• Strong ownership mindset and commitment to engineering excellence.