Azure Databricks Data Engineer

WTW; Kolkata District; On-site; INR 900,000 - 1,500,000; Full time

Job summary

WTW Kolkata is seeking a data engineering specialist to build scalable data pipelines using Azure Databricks, Python, and PySpark. You will orchestrate data movement with ADF pipelines and maintain governance through Unity Catalog.

Ingesting from SQL Server and Oracle, you will transform large datasets and ensure robust, scalable solutions across cloud environments. The role requires 5+ years in data engineering, strong Python skills, and hands-on experience with Azure data services.

Qualifications

• 5+ years of Data Engineering, Data Warehousing experience.
• Bachelor's degree in CS/IT or related field.
• Hands-on Azure Databricks, Delta Lake, and Unity Catalog in enterprise environments.
• Strong Python and PySpark for distributed processing.
• Experience with Azure Data Factory pipelines and ETL/ELT design.

Responsibilities

• Build and maintain scalable data pipelines on Azure Databricks.
• Ingest and transform large datasets from SQL Server/Oracle.
• Manage data governance and access control with Unity Catalog.
• Design and optimize ETL/ELT workflows with ADF.
• Participate in on-call rotation and production support.
• Document runbooks and maintain documentation.

Skills

Python

PySpark

ETL/ELT design

SQL (T-SQL)

CI/CD & DevOps

Distributed data processing

Linux/Windows

Azure cloud services

Education

Bachelor's degree in Computer Science/IT or related field

Tools

Azure Databricks

Delta Lake

Unity Catalog

Azure Data Factory

GitHub Actions

Job description

• Build and maintain scalable pipelines with Azure Databricks, Python, and PySpark.
• Orchestrate data movement and workflows with ADF pipelines, triggers, and integration runtimes.
• Ingest and transform large datasets from sources such as SQL Server and Oracle.
• Manage data governance and access control with Unity Catalog.
• Implement cloud-native data solutions following best practices, with experience in the Azure or AWS cloud.
• Monitor, troubleshoot, and tune pipelines for performance and reliability; take part in the on-call rotation.
• Turn business requirements into robust data solutions and support production releases.
• Maintain documentation and runbooks.

Qualifications:

Graduate

• Minimum 5 years of experience in Data Engineering, Data Warehousing, or related areas.
• Bachelors degree in Computer Science, Information Technology, or a related discipline or equivalent practical experience.
• Strong hands-on experience with Azure Databricks (jobs/workflows, Delta Lake, Unity Catalog) in enterprise-scale environments.
• Advanced Python skills and experience developing distributed data processing solutions using PySpark.
• Hands-on experience with Azure Data Factory pipelines and ETL/ELT design, development, and optimization.
• Strong understanding of Azure cloud services, networking and storage, data modeling, and SQL (T-SQL preferred).
• Working knowledge of Windows and Linux environments.
• Experience with CI/CD and DevOps practices for automated deployment of data solutions (GitHub Actions preferred).
• Proven ability to troubleshoot complex data processing and pipeline issues in production environments.