Azure Senior Data Engineer - S

Tata Consultancy Services; Chennai District, Bengaluru, Kolkata District; On-site; INR 2,500,000 - 4,200,000; Full time

Job summary

Tata Consultancy Services in Chennai seeks a senior Data Engineer to design, develop, and optimize Azure-based data platforms. You will build ETL/ELT pipelines and implement data lake and data warehouse solutions to enable analytics.

Requirements: 6+ years in data engineering with Azure, strong Python/SQL/PySpark skills, and hands-on experience with ADF, Databricks, Synapse, ADLS Gen2, and Cosmos DB. You will mentor juniors, ensure data quality and governance, and drive cost-efficient data

Qualifications

• Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field.
• 6+ years of experience in Data Engineering with at least 3+ years of Azure cloud experience.
• Strong programming skills in: Python, SQL, PySpark, Scala (preferred).
• Hands-on experience with Azure Data Services: Azure Data Factory (ADF), Azure Databricks, Azure Synapse Analytics, Azure Data Lake Storage (ADLS Gen2), Azure SQL Database, Azure Cosmos DB, Azure Event Hubs, Azure Stream Analytics, Azure Functions, Azure Key Vault, Azure Monitor.
• Strong expertise in: ETL/ELT Development, Data Modeling, Data Warehousing, Data Lake Architecture, Dimensional Modeling, Data Governance.
• Experience with Apache Spark and distributed data processing frameworks.
• Strong knowledge of database technologies including SQL and NoSQL databases.
• Experience with version control systems such as Git.
• Excellent analytical, troubleshooting, and problem-solving skills.

Responsibilities

• Design, develop, and maintain scalable and secure data platforms on Microsoft Azure.
• Build and optimize ETL/ELT pipelines for ingesting, transforming, and processing large volumes of data.
• Design and implement Azure-based data lake, data warehouse, and data integration solutions.
• Develop batch and real-time data processing pipelines.
• Integrate data from multiple structured, semi-structured, and unstructured data sources.
• Build and maintain scalable data models to support analytics and reporting needs.
• Ensure data quality, governance, security, and compliance standards are implemented.
• Optimize data pipeline performance, reliability, and cost efficiency.
• Collaborate with Data Scientists, Analysts, Architects, and business stakeholders to understand data requirements.
• Implement monitoring, logging, and alerting for data pipelines and platforms.
• Troubleshoot and resolve production data issues and bottlenecks.
• Support cloud migration and modernization initiatives.
• Develop technical documentation, architecture diagrams, and operational runbooks.
• Mentor junior engineers and promote best practices in data engineering.

Skills

Python

SQL

PySpark

Scala

Education

Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field

Tools

Azure Data Factory (ADF)

Azure Databricks

Azure Synapse Analytics

Azure Data Lake Storage (ADLS Gen2)

Azure SQL Database

Azure Cosmos DB

Azure Event Hubs

Azure Stream Analytics

Azure Functions

Azure Key Vault

Azure Monitor

Git

Job description

Key Responsibilities

• Design, develop, and maintain scalable and secure data platforms on Microsoft Azure.


• Build and optimize ETL/ELT pipelines for ingesting, transforming, and processing large volumes of data.


• Design and implement Azure-based data lake, data warehouse, and data integration solutions.


• Develop batch and real-time data processing pipelines.


• Integrate data from multiple structured, semi-structured, and unstructured data sources.


• Build and maintain scalable data models to support analytics and reporting needs.


• Ensure data quality, governance, security, and compliance standards are implemented.


• Optimize data pipeline performance, reliability, and cost efficiency.


• Collaborate with Data Scientists, Analysts, Architects, and business stakeholders to understand data requirements.


• Implement monitoring, logging, and alerting for data pipelines and platforms.


• Troubleshoot and resolve production data issues and bottlenecks.


• Support cloud migration and modernization initiatives.


• Develop technical documentation, architecture diagrams, and operational runbooks.


• Mentor junior engineers and promote best practices in data engineering.

Required Skills & Qualifications

• Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field.


• 6+ years of experience in Data Engineering with at least 3+ years of Azure cloud experience.


• Strong programming skills in:




• Python


• SQL


• PySpark


• Scala (preferred)




• Hands-on experience with Azure Data Services:




• Azure Data Factory (ADF)


• Azure Databricks


• Azure Synapse Analytics


• Azure Data Lake Storage (ADLS Gen2)


• Azure SQL Database


• Azure Cosmos DB


• Azure Event Hubs


• Azure Stream Analytics


• Azure Functions


• Azure Key Vault


• Azure Monitor




• Strong expertise in:




• ETL/ELT Development


• Data Modeling


• Data Warehousing


• Data Lake Architecture


• Dimensional Modeling


• Data Governance




• Experience with Apache Spark and distributed data processing frameworks.


• Strong knowledge of database technologies including SQL and NoSQL databases.


• Experience with version control systems such as Git.


• Excellent analytical, troubleshooting, and problem-solving skills.