Aws Data Engineer (6 years To 12 years)

Tata Consultancy Services; Mumbai; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

Tata Consultancy Services in Mumbai seeks an experienced Data Engineer to design, build, and optimize scalable data pipelines on AWS. You will work with PySpark, AWS Glue, EMR, and implement data ingestion, lakehouse architectures, and governance across large datasets.

Responsibilities include data modeling, pipeline monitoring, and collaborating in Agile cross-functional teams to deliver robust data solutions.

Qualifications

• AWS Services: S3, Glue, EMR, Lambda, Athena, Redshift, CloudWatch, IAM, Step Functions.
• Data Lakes, Data Warehousing, ETL/ELT Development, Batch Processing, Data Modeling, Data Migration.
• Apache Spark, PySpark, Spark SQL.
• Python, SQL, Shell scripting.
• Git, CI/CD Pipelines, Infrastructure as Code (Terraform preferred).
• Databricks, Snowflake, Kafka, Airflow, Hadoop Ecosystem, Docker, Kubernetes, Delta Lake, Streaming Data Pipelines.

Responsibilities

• Design, develop, and maintain scalable data pipelines on AWS.
• Build and optimize ETL/ELT workflows using PySpark, AWS Glue, and EMR.
• Develop data ingestion frameworks for structured and unstructured data.
• Design and implement Data Lake and Data Warehouse solutions.
• Process large-scale datasets using Spark and distributed computing techniques.
• Create and maintain data models supporting reporting and analytics.
• Ensure data quality, governance, security, and compliance standards.
• Monitor, troubleshoot, and optimize production data pipelines.
• Implement CI/CD practices for data engineering solutions.
• Collaborate with cross-functional teams in Agile environments.

Skills

AWS

Data Lakes

Data Warehousing

ETL/ELT

Batch Processing

Data Modeling

Data Migration

Spark / PySpark

Python

SQL

Shell Scripting

DevOps

Tools

Databricks

Snowflake

Kafka

Airflow

Hadoop Ecosystem

Docker

Kubernetes

Delta Lake

Streaming Data Pipelines

Job description

Key Responsibilities:

• Design, develop, and maintain scalable data pipelines on AWS.
• Build and optimize ETL/ELT workflows using PySpark, AWS Glue, and EMR.
• Develop data ingestion frameworks for structured and unstructured data.
• Design and implement Data Lake and Data Warehouse solutions.
• Process large-scale datasets using Spark and distributed computing techniques.
• Create and maintain data models supporting reporting and analytics.
• Ensure data quality, governance, security, and compliance standards.
• Monitor, troubleshoot, and optimize production data pipelines.
• Implement CI/CD practices for data engineering solutions.
• Collaborate with cross-functional teams in Agile environments.

Mandatory Technical Skills:

1) AWS Services: Amazon S3, AWS Glue, EMR, Lambda, Athena, Redshift, CloudWatch, IAM, Step Functions

2) Data Engineering: Data Lakes, Data Warehousing, ETL / ELT Development, Batch Processing, Data Modeling, Data Migration

3) Big Data Technologies: Apache Spark, PySpark, Spark SQL

4) Programming: Python, SQL, Shell Scripting

5) DevOps & Automation: Git, CI/CD Pipelines, Infrastructure as Code (Terraform preferred)

Good-to-Have Skills:

• Databricks
• Snowflake
• Kafka
• Airflow
• Hadoop Ecosystem
• Docker
• Kubernetes
• Delta Lake
• Streaming Data Pipelines
• BFSI Domain Experience

Desired Technical Competencies:

• Distributed Data Processing
• Performance Tuning
• Cloud Architecture
• Data Security & Governance
• Data Integration Patterns