Azure Databricks Data Engineer

WTW group; Kolkata District; On-site; INR 1,800,000 - 3,400,000; Full time

Job summary

WTW group in Kolkata is hiring a data engineer to build and optimize scalable data pipelines using Azure Databricks, PySpark and Python. You will ingest data from SQL Server and Oracle and ensure governance through Unity Catalog in cloud environments.

You will implement Azure Data Factory workflows, monitor performance, troubleshoot production pipelines, and maintain runbooks and documentation. This role requires 5+ years in data engineering and strong SQL knowledge, with CI/CD experience using

Qualifications

• Minimum 5 years of experience in Data Engineering, Data Warehousing, or related areas.
• Bachelor's degree in Computer Science, Information Technology, or a related discipline or equivalent practical experience.
• Strong hands-on experience with Azure Databricks (jobs/workflows, Delta Lake, Unity Catalog) in enterprise-scale environments.
• Advanced Python skills and experience developing distributed data processing solutions using PySpark.
• Hands-on experience with Azure Data Factory pipelines and ETL/ELT design, development, and optimization.
• Strong understanding of Azure cloud services, networking and storage, data modeling, and SQL (T-SQL preferred).
• Working knowledge of Windows and Linux environments.
• Experience with CI/CD and DevOps practices for automated deployment of data solutions (GitHub Actions preferred).
• Proven ability to troubleshoot complex data processing and pipeline issues in production environments.

Responsibilities

• Build and maintain scalable pipelines with Azure Databricks, Python, and PySpark.
• Orchestrate data movement and workflows with ADF pipelines, triggers, and integration runtimes.
• Ingest and transform large datasets from sources such as SQL Server and Oracle.
• Manage data governance and access control with Unity Catalog.
• Implement cloud-native data solutions following best practices, with experience in the Azure or AWS cloud.
• Monitor, troubleshoot, and tune pipelines for performance and reliability; take part in the on-call rotation.
• Turn business requirements into robust data solutions and support production releases.
• Maintain documentation and runbooks.

Skills

Distributed data processing

Troubleshooting

Production support

CI/CD practices

GitHub Actions

Windows

Linux

Education

Bachelor's degree in Computer Science/Information Technology

Graduate or equivalent practical experience

Tools

Azure Databricks

PySpark

Python

Azure Data Factory

Delta Lake

Unity Catalog

GitHub Actions

Windows

Linux

SQL

Job description

• Build and maintain scalable pipelines with Azure Databricks, Python, and PySpark.
• Orchestrate data movement and workflows with ADF pipelines, triggers, and integration runtimes.
• Ingest and transform large datasets from sources such as SQL Server and Oracle.
• Manage data governance and access control with Unity Catalog.
• Implement cloud-native data solutions following best practices, with experience in the Azure or AWS cloud.
• Monitor, troubleshoot, and tune pipelines for performance and reliability; take part in the on-call rotation.
• Turn business requirements into robust data solutions and support production releases.
• Maintain documentation and runbooks.

Qualifications:

Graduate

• Minimum 5 years of experience in Data Engineering, Data Warehousing, or related areas.
• Bachelors degree in Computer Science, Information Technology, or a related discipline or equivalent practical experience.
• Strong hands-on experience with Azure Databricks (jobs/workflows, Delta Lake, Unity Catalog) in enterprise-scale environments.
• Advanced Python skills and experience developing distributed data processing solutions using PySpark.
• Hands-on experience with Azure Data Factory pipelines and ETL/ELT design, development, and optimization.
• Strong understanding of Azure cloud services, networking and storage, data modeling, and SQL (T-SQL preferred).
• Working knowledge of Windows and Linux environments.
• Experience with CI/CD and DevOps practices for automated deployment of data solutions (GitHub Actions preferred).
• Proven ability to troubleshoot complex data processing and pipeline issues in production environments.