AWS Data Engineer --Python Pyspark

Tata Consultancy Services; Kolkata District; On-site; INR 900,000 - 1,500,000; Full time

Job summary

Tata Consultancy Services is seeking a data engineer to design, build, and maintain data transformation pipelines using AWS services in Python, PySpark, and Databricks. The role involves handling large datasets and managing SQL databases like PostgreSQL or MongoDB.

You will collaborate with software engineers, participate in code reviews, and contribute to automation, testing, and deployment efforts across multiple locations in India, including Kolkata.

Qualifications

• Experience in designing data pipelines using AWS services (Lambda Functions, Step Functions) in Python.
• Experience with PySpark and Databricks and building complex data processing engines.

Responsibilities

• Design, build, and maintain data transformation pipelines using AWS services (Lambda Functions, Step Functions) in Python.
• Design, build, and maintain data transformation pipelines using PySpark and Databricks. Implement and maintain complex calculation engine.
• Manage efficiently large datasets.
• Manage SQL databases - PostgreSQL or MongoDB.
• Implement and maintain complex data calculation formulas.
• Open to collaborate with Software Engineers.
• Participate in code reviews, design discussions, and software planning.
• Contribute to automation, testing, and deployment processes.

Skills

Data engineering

AWS

Python

PySpark

Databricks

SQL databases

Tools

Lambda Functions

Step Functions

Databricks

PySpark

PostgreSQL

MongoDB

Job description

Exp must 6 to 12 --Location Bangalore, Pune, Kolkata, Delhi/NCR

Job Requirements*

The successful candidate will be responsible for

• Design, build, and maintain data transformation pipelines using AWS services (Lambda Functions, Step Functions .. ) in Python.
• Design, build, and maintain data transformation pipelines using PySpark and Databricks. Implement and maintain complex calculation engine.
• Manage efficiently large dataset.
• Manage SQL database - Postgresql or MongoDB.
• Implement and maintain complex data calculation formula.
• Open to collaborate with Software Engineers.
• Participate in code reviews, design discussions, and software planning.
• Contribute to automation, testing, and deployment processes.

Key Responsibilities*

• Design, build, and maintain data transformation pipelines using AWS services (Lambda Functions, Step Functions .. ) in Python.
• Design, build, and maintain data transformation pipelines using PySpark and Databricks. Implement and maintain complex calculation engine.
• Manage effeciently large dataset.
• Manage SQL database - Postgresql or MongoDB.
• Implement and maintain complex data calculation formula.
• Open to collaborate with Software Engineers.
• Participate in code reviews, design discussions, and software planning.
• Contribute to automation, testing, and deployment processes.