Senior Data Engineer: Azure Databricks & Data Pipelines

EPAM Systems; Hong Kong; On-site; HKD 600,000 - 1,000,000; Full time

Job summary

EPAM Systems in Hong Kong seeks a Senior Data Engineer to design and build cloud-based data products for insurance analytics. You will create reliable pipelines on Microsoft Azure using Azure Databricks and Python, while improving data quality and governance to deliver trusted insights.

Responsibilities include ETL/ELT development in Azure Databricks, batch and streaming solutions with ADF, Kafka and Event Hubs, and collaboration with stakeholders to define requirements.

Qualifications

• Experience designing data pipelines on Azure Databricks and Spark.
• Proficiency with Azure data services (ADF, Data Lake Gen2) and Spark.
• Clear communication in Cantonese, Chinese and English.

Responsibilities

• Design, build and maintain scalable ETL/ELT pipelines in Azure Databricks.
• Develop batch and streaming data solutions using Azure Data Factory, Data Lake Gen2, Event Hubs and Kafka.
• Collaborate with data architects and stakeholders to define requirements and deliver data assets.
• Implement data quality checks, governance, metadata and lineage tooling (Informatica).
• Apply security controls including RBAC, encryption and secrets management (Azure Key Vault).
• Improve deployment reliability with GitHub Actions automation.
• Troubleshoot pipeline issues and perform root cause analysis.
• Maintain documentation for transformations, testing and runbooks.

Skills

Azure Databricks

Apache Spark

Python

Data governance

RBAC security

GitHub Actions

Stakeholder communication

Streaming data

Tools

Azure Data Factory

Azure Data Lake Storage Gen2

Azure Event Hubs

Apache Kafka

Informatica

Azure Key Vault

Job description

EPAM Systems in Hong Kong seeks a Senior Data Engineer to design and build cloud-based data products for insurance analytics. You will create reliable pipelines on Microsoft Azure using Azure Databricks and Python, while improving data quality and governance to deliver trusted insights.

Responsibilities include ETL/ELT development in Azure Databricks, batch and streaming solutions with ADF, Kafka and Event Hubs, and collaboration with stakeholders to define requirements.