AWS Data Engineer

Tata Consultancy Services; Bengaluru; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

Tata Consultancy Services in Bengaluru seeks a data engineer to design, build, and maintain data pipelines using AWS services (Lambda Functions, Step Functions) in Python. You will work with PySpark and Databricks to implement complex calculation engines, handling large datasets and integrating with SQL databases.

Collaborating with software engineers, you will participate in code reviews, design discussions, and contribute to automation, testing, and deployment processes.

Qualifications

• Experience designing data pipelines on AWS and Python.
• Experience with PySpark and Databricks.
• Experience handling large datasets.
• Collaborate with software engineers and participate in code reviews.

Responsibilities

• Participate in code reviews, design discussions, and software planning.
• Contribute to automation, testing, and deployment processes.

Skills

Python

AWS

PySpark

Databricks

SQL

Tools

PostgreSQL

MongoDB

Job description

• Design, build, and maintain data transformation pipelines using AWS services (Lambda Functions, Step Functions .. ) in Python.
• Design, build, and maintain data transformation pipelines using PySpark and Databricks. Implement and maintain complex calculation engine.
• Manage effeciently large dataset.
• Manage SQL database - Postgresql or MongoDB.
• Implement and maintain complex data calculation formula.
• Open to collaborate with Software Engineers.
• Participate in code reviews, design discussions, and software planning.
• Contribute to automation, testing, and deployment processes.