Azure Data Engineer

Tata Consultancy Services; Chennai District; On-site; INR 900,000 - 1,400,000; Full time

Job summary

Tata Consultancy Services in Chennai, India, is seeking a data engineering professional to design, develop and operate scalable data pipelines using Azure Data Factory, Databricks, Synapse, and Delta Lake. The role emphasizes hands-on coding with Python, PySpark, and SQL for large-scale data processing and analytics.

You will optimize pipelines, collaborate with analysts, and ensure data governance and quality across data lakes and warehouses.

Qualifications

• Hands-on coding experience with Azure Data Factory, Databricks, Synapse and Delta Lake.
• Implement ETL to load data into data lake and data warehouse.
• Proficient in Python, PySpark and SQL for large-scale data processing.
• Optimize and tune data pipelines for performance and scalability.
• Write complex SQL queries.
• Collaborate with business stakeholders to ensure data quality.
• Knowledge of Agile methodologies and DevOps culture.
• Strong problem-solving skills.

Responsibilities

• Design, develop, and deploy scalable data pipelines using Databricks (PySpark, Spark SQL), Azure Synapse, Azure Data Factory, and other Azure data services.
• Implement ETL/ELT processes to ingest, transform, and load data from various sources into data lakes and data warehouses.
• Optimize and tune data pipelines for performance and scalability.
• Write and optimize complex SQL queries for data extraction, transformation, and analysis.
• Use PySpark for large-scale data processing and analytics.
• Implement data partitioning, bucketing, z-ordering, liquid clustering and indexing strategies.
• Integrate data from multiple sources, including structured, semi-structured, and unstructured data.
• Work with APIs, streaming data, and batch processing to ensure seamless data integration.
• Implement data governance practices to ensure data quality, consistency, and security.
• Monitor and troubleshoot data pipelines to ensure data accuracy and availability.
• Collaborate with data scientists, analysts, and stakeholders to understand data requirements.
• Work with DevOps teams to deploy and monitor data pipelines in production.
• Document data pipelines, workflows, and processes for knowledge sharing.
• Maintain up-to-date documentation on data architecture and data models.

Skills

Azure Data Factory

Databricks

Azure Synapse

Delta Lake

Python

PySpark

SQL

ETL

Agile Methodologies

DevOps Culture

Data Pipeline Optimization

Problem Solving

Tools

Spark SQL

Job description

Job Requirements*

• Azure data factory, Databricks, synapse, delta lake development with hands-on coding experience
• Implement ETL solution to integrate, transform and load data from various sources into data lake and data warehouse
• Hands-on expertise in python, pyspark and sql for large scale data processing
• Optimize and tune data pipelines for performance and scalability
• Ability to write complex SQL queries
• Collaborate with business analysts and business stakeholders to gather requirements and ensure data quality and availability.
• Good understanding of Agile Methodologies and DevOps Culture
• Strong Problem-solving skills

.

Key Responsibilities*

• Design, develop, and deploy scalable data pipelines using Databricks (PySpark, Spark SQL), Azure Synapse, Azure Data Factory, and other Azure data services.
• Implement ETL/ELT processes to ingest, transform, and load data from various sources into data lakes and data warehouses.
• Optimize and tune data pipelines for performance and scalability.
• Write and optimize complex SQL queries for data extraction, transformation, and analysis.
• Use PySpark for large-scale data processing and analytics.
• Implement data partitioning, bucketing, z-ordering, liquid clustering and indexing strategies for efficient data retrieval.
• Integrate data from multiple sources, including structured, semi-structured, and unstructured data.
• Work with APIs, streaming data, and batch processing to ensure seamless data integration.
• Implement data governance practices to ensure data quality, consistency, and security.
• Monitor and troubleshoot data pipelines to ensure data accuracy and availability.
• Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions.
• Work closely with DevOps teams to deploy and monitor data pipelines in production environments.
• Document data pipelines, workflows, and processes for knowledge sharing and future reference.
• Maintain up-to-date documentation on data architecture and data models.