AWS Data Engineer -Python Pyspark

Tata Consultancy Services; Kolkata District; On-site; INR 900,000 - 1,300,000; Full time

Job summary

Tata Consultancy Services seeks a data engineering specialist for Kolkata to design, build, and maintain data pipelines using AWS and PySpark. You will work with Databricks, handle large datasets, and manage PostgreSQL or MongoDB databases.

You will collaborate with software engineers, participate in code reviews, and contribute to automation, testing, and deployment efforts to deliver scalable data solutions.

Qualifications

• Design, build, and maintain data pipelines using AWS services (Lambda Functions, Step Functions) in Python.
• Design, build, and maintain data transformation pipelines using PySpark and Databricks. Implement and maintain complex calculation engine.
• Manage efficiently large dataset.
• Manage SQL database - PostgreSQL or MongoDB.

Responsibilities

• Design, build, and maintain data transformation pipelines using AWS services (Lambda Functions, Step Functions .. ) in Python.
• Design, build, and maintain data transformation pipelines using PySpark and Databricks. Implement and maintain complex calculation engine.
• Manage effeciently large dataset.
• Manage SQL database - Postgresql or MongoDB.
• Implement and maintain complex data calculation formula.
• Open to collaborate with Software Engineers.
• Participate in code reviews, design discussions, and software planning.
• Contribute to automation, testing, and deployment processes.

Skills

AWS

Python

PySpark

Databricks

Data pipelines

SQL databases

PostgreSQL

MongoDB

Large datasets

Collaboration

Tools

PostgreSQL

MongoDB

Git

Job description

Exp must 6 to 12 --Location Bangalore, Pune, Kolkata, Delhi/NCR

Job Requirements*

• Design, build, and maintain data transformation pipelines using AWS services (Lambda Functions, Step Functions .. ) in Python.
• Design, build, and maintain data transformation pipelines using PySpark and Databricks. Implement and maintain complex calculation engine.
• Manage efficiently large dataset.
• Manage SQL database - Postgresql or MongoDB.
• Implement and maintain complex data calculation formula.
• Open to collaborate with Software Engineers.
• Participate in code reviews, design discussions, and software planning.
• Contribute to automation, testing, and deployment processes.

Key Responsibilities*

• Design, build, and maintain data transformation pipelines using AWS services (Lambda Functions, Step Functions .. ) in Python.
• Design, build, and maintain data transformation pipelines using PySpark and Databricks. Implement and maintain complex calculation engine.
• Manage effeciently large dataset.
• Manage SQL database - Postgresql or MongoDB.
• Implement and maintain complex data calculation formula.
• Open to collaborate with Software Engineers.
• Participate in code reviews, design discussions, and software planning.
• Contribute to automation, testing, and deployment processes.