Data Engineer -Kolkata - 8th Sep (Tuesday)-8+Years- Video Interview

Tata Consultancy Services; Kolkata District; On-site; INR 1,800,000 - 3,800,000; Full time

Job summary

Tata Consultancy Services in Kolkata is seeking an experienced Senior Data Engineer to design, build and optimize scalable data pipelines using Azure Data Factory, Databricks, and Delta Live Tables. You will work closely with data teams to translate business needs into robust data solutions.

Strong Python/SQL skills, Terraform, YAML, and governance tools like Purview or Informatica are essential. You will implement CI/CD for data workflows, ensure data quality and security, and collaborate with

Qualifications

• Experience designing scalable data pipelines and data architectures.
• Proficiency with Azure Data Factory, Databricks, and Delta Live Tables.
• Hands-on coding with Python, SQL, YAML and Terraform.
• Knowledge of data governance, security, and catalog tools like Purview or Informatica.

Responsibilities

• Design, build, and optimize scalable data pipelines using Azure Data Factory, Databricks, and Delta Live Tables.
• Implement ETL processes across diverse data assets ensuring data quality and performance.
• Develop and manage data warehouses and lakehouses for structured and semi-structured data.
• Ensure compliance with data security, governance, and privacy standards using Purview and Informatica.
• Collaborate with data scientists and ML engineers to operationalize machine learning pipelines.
• Implement CI/CD pipelines for automating data pipeline deployments.
• Translate complex technical solutions into business insights for stakeholders.
• Troubleshoot and resolve performance bottlenecks in data workflows.

Skills

Data pipelines

Python

SQL

Data modelling

Tools

Azure Data Factory

Databricks

Delta Live Tables

Terraform

YAML

Purview

Informatica

Azure Services

CI/CD

Data Warehousing

Job description

Senior Data Engineer

Azure Data Factory, Databricks, Delta Live Tables, Python, SQL, Terraform, YAML, Azure Services, Data Warehousing, CI/CD, Data Governance, Purview, Informatica

Strong proficiency in Azure Data Factory, Databricks, Delta Live Tables • Solid experience with Python, SQL, YAML, and Terraform • Hands-on experience in building and optimizing scalable data pipelines • Expertise in Data Warehousing, Lakehouse Architecture, and Data Modelling • Knowledge of CI/CD practices and data governance tools like Purview or Informatica

Design, build, and optimize scalable data pipelines using Azure Data Factory, Databricks, and Delta Live Tables.

2 Implement ETL processes across diverse data assets ensuring data quality and performance.

3 Develop and manage data warehouses and lakehouses for structured and semi-structured data.

4 Ensure compliance with data security, governance, and privacy standards using Purview and Informatica.

5 Collaborate with data scientists and ML engineers to operationalize machine learning pipelines.

6 Implement CI/CD pipelines for automating data pipeline deployments.

7 Translate complex technical solutions into business insights for stakeholders.

8 Troubleshoot and resolve performance bottlenecks in data workflows.