AWS Cloud, AWS EMR, Amazon S3, Python, Scala, Data Engineering

Tata Consultancy Services; Bengaluru; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

TCS is hiring for a data engineering role focused on AWS cloud, PySpark, and Scala. The candidate will work on building scalable data pipelines, leveraging Spark ecosystem and AWS services to design and optimize ETL processes in a Hadoop ecosystem.

Ideal candidates have strong Python and PySpark skills, hands-on AWS experience (Redshift/EMR/S3/Glue), SQL proficiency, and familiarity with Airflow for scheduling. Minimum 8+ years IT experience required.

Qualifications

• Strong hands-on experience in Python and PySpark.
• Experience with AWS services including RedShift, Glue, EMR, S3, and Lambda.
• Scala experience with Python/PySpark for building scalable data engineering solutions on AWS/Spark ecosystems.
• Experience with Apache Spark and Hadoop ecosystem.
• SQL for data manipulations.
• Exposure to scheduling tools, Airflow preferred.
• Data Warehouse experience with AWS Redshift or Hive.
• Experience implementing security measures for data protection.
• Ability to build/test complex ETL pipelines (batch and near real time).
• Readable documentation of all developed components.

Skills

Python

PySpark

AWS Redshift

AWS EMR

AWS Glue

Scala

Spark

SQL

Airflow

Data Warehouse

Hive

Security

ETL pipelines

Documentation

OLTP/OLAP

Education

15 years education

Tools

NoSQL databases

DynamoDB

MongoDB

Step Functions

Athena

Job description

TCS Hiring for AWS Cloud, AWS EMR, Amazon S3, Python, Scala, Data Engineering

Experience Range: - 08 To 15 Years (Mandatory)

(Note: Candidates below 8 years of IT experience shall not be considered)

JOB DESCRIPTION

Must Have Skills:

• Strong hands-on experience in Python programming and PySpark
• Experience using AWS services (RedShift, Glue, EMR, S3 & Lambda)
• Hands-on experience in Scala along with Python/PySpark is strongly preferred for building scalable data engineering solutions on AWS and Spark ecosystems.
• Experience working with Apache Spark and Hadoop ecosystem.
• Experience in writing and optimizing SQL for data manipulations.
• Good Exposure to scheduling tools. Airflow is preferable.
• Must – Have Data Warehouse Experience with AWS Redshift or Hive
• Experience in implementing security measures for data protection.
• Expertise to build/test complex data pipelines for ETL processes (batch and near real time)
• Readable documentation of all the components being developed.
• Knowledge of Database technologies for OLTP and OLAP workloads

Good to Have Skills:

• Good understanding of Data warehouse and Data Lakes
• Experience working with NoSQL databases like DynamoDB or MongoDB.
• Good to have exposure to AWS services (Step Functions, Athena)
• Familiarity with Investment Banking domain

15 years of Full Time Education