Azure Databricks Data Engineer

WTW Global Delivery And Solutions; Kolkata District; Hybrid; INR 1,200,000 - 2,600,000; Full time

Job summary

WTW Global Delivery And Solutions is seeking an experienced Data Engineer in Kolkata to design and operate scalable data pipelines using Azure Databricks, PySpark, and Python. The role focuses on building robust ETL/ELT processes with ADF, managing data governance via Unity Catalog, and ensuring performance and reliability in enterprise environments.

The ideal candidate brings 5+ years in data engineering or warehousing, strong SQL skills, and hands-on Azure cloud experience.

Qualifications

• 5+ years of experience in Data Engineering, Data Warehousing, or related areas.
• Bachelor's degree in Computer Science, Information Technology, or a related discipline or equivalent practical experience.
• Strong hands-on experience with Azure Databricks (jobs/workflows, Delta Lake, Unity Catalog) in enterprise-scale environments.
• Advanced Python skills and experience developing distributed data processing solutions using PySpark.
• Hands-on experience with Azure Data Factory pipelines and ETL/ELT design, development, and optimization.
• Strong understanding of Azure cloud services, networking and storage, data modeling, and SQL (T-SQL preferred).
• Working knowledge of Windows and Linux environments.
• Experience with CI/CD and DevOps practices for automated deployment of data solutions (GitHub Actions preferred).
• Proven ability to troubleshoot complex data processing and pipeline issues in production environments.

Responsibilities

• Build and maintain scalable pipelines with Azure Databricks, Python, and PySpark.
• Orchestrate data movement and workflows with ADF pipelines, triggers, and integration runtimes.
• Ingest and transform large datasets from sources such as SQL Server and Oracle.
• Manage data governance and access control with Unity Catalog.
• Implement cloud-native data solutions following best practices, with experience in the Azure or AWS cloud.
• Monitor, troubleshoot, and tune pipelines for performance and reliability; take part in the on-call rotation.
• Turn business requirements into robust data solutions and support production releases.
• Maintain documentation and runbooks.

Skills

Data engineering

Data warehousing

Distributed processing

Troubleshooting

DevOps practices

Education

Bachelor's degree in CS/IT or related field

Tools

Azure Databricks

Python

PySpark

Azure Data Factory

Delta Lake

Unity Catalog

SQL Server

Oracle

Windows

Linux

GitHub Actions

Azure

AWS

Job description

Role and Responsibility:

• Build and maintain scalable pipelines with Azure Databricks, Python, and PySpark.
• Orchestrate data movement and workflows with ADF pipelines, triggers, and integration runtimes.
• Ingest and transform large datasets from sources such as SQL Server and Oracle.
• Manage data governance and access control with Unity Catalog.
• Implement cloud-native data solutions following best practices, with experience in the Azure or AWS cloud.
• Monitor, troubleshoot, and tune pipelines for performance and reliability; take part in the on-call rotation.
• Turn business requirements into robust data solutions and support production releases.
• Maintain documentation and runbooks.

Qualifications:

Graduate

• Minimum 5 years of experience in Data Engineering, Data Warehousing, or related areas.
• Bachelors degree in Computer Science, Information Technology, or a related discipline or equivalent practical experience.
• Strong hands-on experience with Azure Databricks (jobs/workflows, Delta Lake, Unity Catalog) in enterprise-scale environments.
• Advanced Python skills and experience developing distributed data processing solutions using PySpark.
• Hands-on experience with Azure Data Factory pipelines and ETL/ELT design, development, and optimization.
• Strong understanding of Azure cloud services, networking and storage, data modeling, and SQL (T-SQL preferred).
• Working knowledge of Windows and Linux environments.
• Experience with CI/CD and DevOps practices for automated deployment of data solutions (GitHub Actions preferred).
• Proven ability to troubleshoot complex data processing and pipeline issues in production environments.