Azure Databricks

Tata Consultancy Services; Pune District; On-site; INR 1,800,000 - 2,800,000; Full time

Job summary

Tata Consultancy Services in Pune is seeking an Azure Databricks Engineer with 6-10 years of experience to design, build, and optimize scalable data pipelines on Azure. The role focuses on PySpark development, ETL/ELT workflows, and end-to-end data solutions for analytics teams.

You will work with Azure Data Factory, Delta Lake, and Synapse to deliver reliable data platforms, collaborate with stakeholders, and contribute to CI/CD practices and production support.

Qualifications

• : 6+ years of data engineering / Azure Databricks experience.

Responsibilities

• Design, develop, and maintain ETL/ELT pipelines using Azure Databricks and PySpark.
• Build scalable data processing solutions for batch and real-time workloads.
• Integrate data from multiple sources including databases, APIs, and cloud storage.
• Develop and optimize Spark jobs for performance and scalability.
• Implement data quality, validation, and monitoring frameworks.
• Work with Azure services to build end-to-end data engineering solutions.
• Collaborate with business teams to understand data requirements and deliver analytics-ready datasets.
• Support CI/CD implementation and deployment automation.
• Troubleshoot production issues and optimize existing data pipelines.

Skills

Azure Databricks

PySpark

Apache Spark

SQL

Azure Data Factory

Delta Lake

Git

Python

Scala

Kafka

Event Hub

Tools

Azure Data Factory

ADLS Gen2

Azure Synapse Analytics

Azure Blob Storage

Azure Key Vault

Azure DevOps

Delta Lake

Git

Job description

Role & responsibilities

Job Title: Azure Databricks Engineer / Senior Azure Databricks Developer

Location: Pune

Experience: 6-10 Years

Preferred candidate profile

Job Summary:

We are looking for a skilled Azure Databricks Engineer with strong expertise in building scalable data pipelines, data transformation, and cloud-based analytics solutions on the Azure platform. The candidate will work closely with data engineers, architects, and business stakeholders to design, develop, and optimize modern data solutions using Azure Databricks, PySpark, and Azure Data Services.

Key Responsibilities:

• Design, develop, and maintain ETL/ELT pipelines using Azure Databricks and PySpark.
• Build scalable data processing solutions for batch and real-time workloads.
• Integrate data from multiple sources including databases, APIs, and cloud storage.
• Develop and optimize Spark jobs for performance and scalability.
• Implement data quality, validation, and monitoring frameworks.
• Work with Azure services to build end-to-end data engineering solutions.
• Collaborate with business teams to understand data requirements and deliver analytics-ready datasets.
• Support CI/CD implementation and deployment automation.
• Troubleshoot production issues and optimize existing data pipelines.

Required Skills:

• 5+ years of Data Engineering experience.
• Strong hands-on experience with Azure Databricks.
• Expertise in PySpark and Apache Spark.
• Strong SQL programming skills.
• Experience with:
• Azure Data Factory (ADF)
• Azure Data Lake Storage (ADLS Gen2)
• Azure Synapse Analytics
• Azure Blob Storage
• Azure Key Vault
• Azure DevOps
• Experience in data modeling and ETL development.
• Knowledge of Delta Lake architecture.
• Understanding of performance tuning and optimization techniques.
• Experience with Git version control.

Preferred Skills:

• Azure Data Engineer Certification (DP-203).
• Experience with Kafka/Event Hub.
• Knowledge of Scala or Python programming.
• Experience in Data Warehousing concepts.
• Exposure to Snowflake is a plus.
• Experience with Agile/Scrum methodologies.