Azure Data Engineer (5 To 10 years)- Chennai/ Bangalore

Tata Consultancy Services; Chennai District, Bengaluru; On-site; INR 900,000 - 1,500,000; Full time

Job summary

Tata Consultancy Services in Chennai invites experienced data engineers to design, develop, and deploy scalable data pipelines using Databricks, Azure Synapse, and Azure Data Factory. You will work with PySpark, Python, and SQL to ingest, transform, and load data into data lakes and warehouses.

You will optimize pipelines for performance, implement governance, collaborate with analysts and data scientists, and align with DevOps practices to ensure reliable production deployments.

Qualifications

• Hands-on experience building ETL/ELT pipelines with Azure data services.
• Strong Python/PySpark and SQL for large-scale data processing.
• Ability to optimize pipelines for performance and scalability.
• Good understanding of Agile methodologies and DevOps culture.

Responsibilities

• Design, develop, and deploy scalable data pipelines using Databricks, Azure Synapse, and Data Factory.
• Implement ETL/ELT processes to ingest, transform, and load data into data lakes and warehouses.
• Optimize pipelines for performance and scalability.
• Write and optimize complex SQL queries for extraction and analysis.
• Use PySpark for large-scale data processing and analytics.
• Implement data partitioning and indexing strategies for efficient retrieval.
• Integrate data from multiple sources and manage APIs/streaming data.
• Enforce data governance to ensure quality, consistency, and security.

Skills

Azure Data Factory

Databricks

Azure Synapse

Delta Lake

ETL Development

Python

PySpark

SQL

Data Quality

Agile / DevOps

Tools

Spark SQL

SQL Server

Azure (Cloud)

Job description

Job Requirements*

• Azure data factory, Databricks, synapse, delta lake development with hands-on coding experience
• Implement ETL solution to integrate, transform and load data from various sources into data lake and data warehouse
• Hands-on expertise in python, pyspark and sql for large scale data processing
• Optimize and tune data pipelines for performance and scalability
• Ability to write complex SQL queries
• Collaborate with business analysts and business stakeholders to gather requirements and ensure data quality and availability.
• Good understanding of Agile Methodologies and DevOps Culture
• Strong Problem-solving skills

.

Key Responsibilities*

• Design, develop, and deploy scalable data pipelines using Databricks (PySpark, Spark SQL), Azure Synapse, Azure Data Factory, and other Azure data services.
• Implement ETL/ELT processes to ingest, transform, and load data from various sources into data lakes and data warehouses.
• Optimize and tune data pipelines for performance and scalability.
• Write and optimize complex SQL queries for data extraction, transformation, and analysis.
• Use PySpark for large-scale data processing and analytics.
• Implement data partitioning, bucketing, z-ordering, liquid clustering and indexing strategies for efficient data retrieval.
• Integrate data from multiple sources, including structured, semi-structured, and unstructured data.
• Work with APIs, streaming data, and batch processing to ensure seamless data integration.
• Implement data governance practices to ensure data quality, consistency, and security.
• Monitor and troubleshoot data pipelines to ensure data accuracy and availability.
• Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions.
• Work closely with DevOps teams to deploy and monitor data pipelines in production environments.
• Document data pipelines, workflows, and processes for knowledge sharing and future reference.
• Maintain up-to-date documentation on data architecture and data models.