AWS Senior Data Engineer

Tata Consultancy Services; Chennai District; On-site; INR 2,500,000 - 4,200,000; Full time

Job summary

Tata Consultancy Services is seeking an experienced AWS Data Engineer to design and optimize large-scale batch and real-time data platforms in Chennai. You will work with PySpark, Spark SQL, Python, and AWS services to build scalable pipelines and streaming solutions.

Strong focus on data quality, governance, and cost efficiency across data lakehouse architectures. Collaboration across teams, establishing CI/CD practices, and troubleshooting performance issues will be core responsibilities for

Qualifications

• Must have 10-15 years of experience in data engineering and cloud data platforms.
• Proficient in batch and real-time data processing using PySpark and Spark SQL.
• Solid experience with AWS data services and Kafka-based streaming.

Responsibilities

• Design and implement scalable batch and streaming data pipelines on AWS.
• Develop high-performance PySpark and Spark SQL applications.
• Build Kafka-based event-driven and real-time streaming solutions.
• Integrate AWS data services for reliability and cost efficiency.
• Apply data quality, governance, monitoring, and security best practices.
• Create CI/CD pipelines and support production deployments.
• Troubleshoot performance bottlenecks and incidents in production.

Skills

PySpark

Spark SQL

Python

SQL

Kafka

AWS

Spark Structured Streaming

Data Modeling

Data Lakes

CI/CD

Git

Jenkins/GitHub Actions

Data Quality

Governance & Security

Education

Tools

Databricks

Delta Lake

Unity Catalog

Airflow / MWAA

Snowflake

DBT

Terraform

Kinesis

AWS MSK

Job description

Experience: 10-15 Years

We are seeking an experienced AWS Data Engineer with strong expertise in Kafka, PySpark, Spark SQL, Python, and AWS to build and optimize large-scale batch and real-time data platforms.

Key Skills

• PySpark, Spark SQL, Python, SQL
• AWS: S3, Glue, EMR, Lambda, IAM, Redshift, Step Functions, CloudWatch
• Spark Structured Streaming
• Data Modeling, Data Lakes & Lakehouse Architecture
• CI/CD, Git, Jenkins/GitHub Actions
• Data Quality, Governance & Security

Responsibilities

• Design and develop scalable batch and streaming data pipelines on AWS.
• Build high-performance PySpark and Spark SQL applications.
• Implement Kafka-based event-driven and real-time streaming solutions.
• Integrate and optimize AWS data services for reliability and cost efficiency.
• Implement data quality, governance, monitoring, and security best practices.
• Build CI/CD pipelines and support production deployments.
• Troubleshoot performance bottlenecks and production issues.

Good to Have

• Amazon MSK, Kinesis
• Databricks, Delta Lake, Unity Catalog
• Airflow / MWAA
• Snowflake, DBT
• Terraform

Primary Skills: AWS Data Engineering, Kafka, PySpark, Spark SQL, Python, Streaming Data Engineering.