AWS Data Engineer --Python Pyspark

Tata Consultancy Services; Bengaluru; Hybrid; INR 1,500,000 - 2,500,000; Full time

Job summary

Tata Consultancy Services in Bengaluru/Pune invites a skilled Data Engineer to design, build, and optimize data transformation pipelines using AWS services (Lambda, Step Functions) in Python. You will handle large datasets and implement robust data processing architectures.

You'll work with PySpark and Databricks, manage PostgreSQL or MongoDB, and collaborate with software engineers, contributing to code reviews, testing, and deployment automation.

Qualifications

• Experience building data transformation pipelines using AWS and Python.
• Proficiency with PySpark and Databricks for large-scale data processing.
• Experience managing large datasets and both SQL and NoSQL databases.

Responsibilities

• Design, build, and maintain data transformation pipelines using AWS services (Lambda Functions, Step Functions) in Python.
• Design, build, and maintain data transformation pipelines using PySpark and Databricks. Implement and maintain complex calculation engine.
• Manage large datasets efficiently and maintain SQL databases such as PostgreSQL or MongoDB.
• Collaborate with Software Engineers and participate in code reviews, design discussions, and deployment processes.

Skills

Python

SQL

Big data concepts

Tools

AWS Lambda

AWS Step Functions

PySpark

Databricks

PostgreSQL

MongoDB

Job description

Exp must 6 to 12 --Location Bangalore, Pune, Virtual Drive on 15th Oct

Job Requirements*

The successful candidate will be responsible for

• Design, build, and maintain data transformation pipelines using AWS services (Lambda Functions, Step Functions .. ) in Python.
• Design, build, and maintain data transformation pipelines using PySpark and Databricks. Implement and maintain complex calculation engine.
• Manage efficiently large dataset.
• Manage SQL database - Postgresql or MongoDB.
• Implement and maintain complex data calculation formula.
• Open to collaborate with Software Engineers.
• Participate in code reviews, design discussions, and software planning.
• Contribute to automation, testing, and deployment processes.

Key Responsibilities*

• Design, build, and maintain data transformation pipelines using AWS services (Lambda Functions, Step Functions .. ) in Python.
• Design, build, and maintain data transformation pipelines using PySpark and Databricks. Implement and maintain complex calculation engine.
• Manage effeciently large dataset.
• Manage SQL database - Postgresql or MongoDB.
• Implement and maintain complex data calculation formula.
• Open to collaborate with Software Engineers.
• Participate in code reviews, design discussions, and software planning.
• Contribute to automation, testing, and deployment processes.