Data Engineer

Tata Consultancy Services; Gurugram District; On-site; INR 1,500,000 - 2,800,000; Full time

Job summary

Tata Consultancy Services in India is seeking an experienced Azure Databricks Developer to design, build, and optimize scalable data solutions on the Azure cloud. You will work with PySpark, Delta Lake, and Lakehouse patterns to deliver robust data pipelines.

You will integrate ADLS Gen2, Azure Data Factory, and SQL data stores, enforce data quality and security, and collaborate with stakeholders to translate business needs into reliable data architectures.

Qualifications

• 7+ years of IT experience with strong Data Engineering background.
• Strong proficiency in PySpark and Spark SQL.
• Experience with Azure Data Factory (ADF).
• Knowledge of Azure Data Lake Storage (ADLS Gen2).
• Experience with Delta Lake and Lakehouse architecture.

Responsibilities

• Develop and maintain ETL/ELT data pipelines using Azure Databricks and PySpark.
• Design and implement scalable data processing solutions for structured and unstructured datasets.
• Build, optimize, and manage Delta Lake architectures.
• Integrate data from multiple sources such as Azure Data Lake Storage, Azure SQL Database, APIs, and third-party systems.
• Collaborate with business stakeholders, data architects, and analytics teams to understand requirements.
• Monitor, troubleshoot, and optimize data workflows for performance and reliability.
• Implement data quality, governance, and security best practices.
• Support data migration and modernization initiatives on Azure.

Skills

PySpark

Spark SQL

Data Engineering

ETL/ELT

Data warehousing concepts

SQL development

Tools

Azure Databricks

Azure Data Factory (ADF)

Azure Data Lake Storage (ADLS Gen2)

Delta Lake

CI/CD pipelines

Git-based source control

Job description

Job Description

We are looking for an experienced Azure Databricks Developer to design, develop, and optimize scalable data solutions on the Azure cloud platform. The ideal candidate should have strong expertise in Azure Databricks, PySpark, Data Engineering, and modern data warehousing concepts.

Key Responsibilities

• Develop and maintain ETL/ELT data pipelines using Azure Databricks and PySpark.
• Design and implement scalable data processing solutions for structured and unstructured datasets.
• Build, optimize, and manage Delta Lake architectures.
• Integrate data from multiple sources such as Azure Data Lake Storage, Azure SQL Database, APIs, and third-party systems.
• Collaborate with business stakeholders, data architects, and analytics teams to understand requirements.
• Monitor, troubleshoot, and optimize data workflows for performance and reliability.
• Implement data quality, governance, and security best practices.
• Support data migration and modernization initiatives on Azure.

Mandatory Skills

• 7+ years of IT experience with strong Data Engineering background.
• Strong proficiency in PySpark and Spark SQL.
• Experience with Azure Data Factory (ADF).
• Knowledge of Azure Data Lake Storage (ADLS Gen2).
• Experience with Delta Lake and Lakehouse architecture.
• Strong SQL development and performance tuning skills.
• Understanding of Data Warehousing concepts and ETL frameworks.
• Experience with Git-based source control and CI/CD pipelines.