Azure Data Engineer - 4th Sep(Friday)-6+ years - Video Interview

Tata Consultancy Services; Pune District, Bengaluru, Delhi; On-site; INR 900,000 - 1,300,000; Full time

Job summary

Tata Consultancy Services in Pune is seeking a data engineer to design, build, and maintain data transformation pipelines using Azure Functions, PySpark, and Databricks.

You will manage large datasets, work with SQL databases like PostgreSQL or MongoDB, and collaborate with software engineers on automation, testing, and deployment.

Qualifications

• Experience designing data pipelines using cloud services and Python.
• Hands-on experience with PySpark and Databricks for large-scale data processing.
• Familiarity with SQL databases such as PostgreSQL or MongoDB.

Responsibilities

• Design, build, and maintain data transformation pipelines using Azure services (Azure Functions, Step Functions) in Python.
• Design, build, and maintain data transformation pipelines using PySpark and Databricks, implementing complex calculation engines.
• Manage efficiently large datasets and ensure data quality and scalability.
• Maintain SQL databases - PostgreSQL or MongoDB, ensure performance and reliability.
• Implement and maintain complex data calculation formulas.
• Collaborate with Software Engineers and participate in code reviews, design discussions, and software planning.
• Contribute to automation, testing, and deployment processes.

Skills

Python

Azure Functions

Step Functions

PySpark

Databricks

Data pipeline design

Large dataset handling

SQL databases

Tools

PostgreSQL

MongoDB

Job description

The successful candidate will be responsible for

• Design, build, and maintain data transformation pipelines using AZURE services (Azure Functions, Step Functions .. ) in Python.
• Design, build, and maintain data transformation pipelines using PySpark and Databricks. Implement and maintain complex calculation engine.
• Manage efficiently large dataset.
• Manage SQL database - Postgresql or MongoDB.
• Implement and maintain complex data calculation formula.
• Open to collaborate with Software Engineers.
• Participate in code reviews, design discussions, and software planning.
• Contribute to automation, testing, and deployment processes.

Key Responsibilities*

• Design, build, and maintain data transformation pipelines using AZURE services (Azure Functions, Step Functions .. ) in Python.
• Design, build, and maintain data transformation pipelines using PySpark and Databricks. Implement and maintain complex calculation engine.
• Manage effeciently large dataset.
• Manage SQL database - Postgresql or MongoDB.
• Implement and maintain complex data calculation formula.
• Open to collaborate with Software Engineers.
• Participate in code reviews, design discussions, and software planning.
• Contribute to automation, testing, and deployment processes.