Data Engineer - Databricks

Don Nelson Recruitment Limited; Hong Kong; On-site; HKD 540,000 - 740,000; Full time

Job summary

Don Nelson Technology Limited seeks a skilled Data Engineer to design and maintain scalable Azure Databricks pipelines for insurance data. You will collaborate with architects, modellers, analysts and stakeholders to deliver robust data assets for analytics, operations, and regulatory reporting.

You will develop batch and streaming solutions using Azure Data Factory, Data Lake Gen2, Event Hubs and Kafka, with strong governance, security, and CI/CD practices.

Qualifications

• Bachelor's degree in CS/IT or related field.
• 3+ years with Azure cloud platforms and Databricks for insurance workloads.
• Experience provisioning and managing Databricks clusters/workspaces.
• Experience integrating Data Factory and Data Lake Gen2 with Databricks.
• Hands-on with Informatica for governance and metadata management.
• Strong Azure security, monitoring and cost optimization in regulated environments.
• CI/CD automation using GitHub Actions for deployment.
• Proven ability to troubleshoot data workloads in insurance.

Responsibilities

• Design and maintain scalable ETL/ELT pipelines in Azure Databricks.
• Collaborate with architects, modellers, analysts and stakeholders.
• Develop and optimize batch and streaming data solutions using ADF, Databricks, Kafka.
• Implement data validation and quality controls for critical use cases.
• Enforce governance, metadata management and cataloguing with Informatica.
• Apply domain expertise in Hong Kong Life and General Insurance.
• Automate deployment workflows with GitHub Actions.
• Troubleshoot pipelines and maintain detailed technical documentation.

Skills

Azure Databricks

Data Integration

Data Governance

Security & Compliance

CI/CD

Cloud Architecture

Education

Bachelor's degree in CS/IT

Tools

Azure Data Factory

Azure Data Lake Storage Gen2

Azure Event Hubs

Kafka

Informatica

Terraform

GitHub Actions

Job description

Design and maintain scalable ETL/ELT pipelines in Azure Databricks to process structured, semi-structured, and unstructured insurance data from diverse sources.

Collaborate with architects, modellers, analysts, and stakeholders to deliver tailored data assets for analytics, operations, and regulatory reporting.

Develop and optimize batch and streaming data solutions using Azure Data Factory, Data Lake Storage Gen2, Azure Event Hubs, and Kafka.

Implement robust data validation, cleansing, and quality controls to ensure reliability for critical insurance use cases.

Integrate Informatica tools to support governance, metadata management, lineage tracking, and cataloguing.

Enforce data security best practices including RBAC, managed identities, encryption, and compliance with PDPO, GDPR, and other regulations.

Automate deployment workflows using GitHub Actions for efficient, repeatable, and auditable data operations.

Troubleshoot pipeline issues, conduct root cause analysis, and proactively resolve data quality and performance challenges.

Maintain detailed technical documentation for data pipelines, transformation logic, and operational procedures.

Apply domain expertise in Hong Kong Life and General Insurance to ensure solutions meet local business and regulatory standards.

Requirements

Bachelor’s degree in Computer Science, IT, or a related field.

3+ years of experience with Azure cloud platforms, focused on Databricks for insurance data workloads.

Strong skills in provisioning and managing Azure Databricks clusters and workspaces for varied data types.

Experience integrating Azure Data Factory and Data Lake Storage Gen2 with Databricks for seamless data flows.

Proficient in Terraform for infrastructure-as-code deployment of Azure and Databricks services.

Hands‑on experience with Informatica for metadata management, cataloguing, and governance.

Deep understanding of Azure security (RBAC, NSG, Key Vault), monitoring, and cost optimization in regulated environments.

Skilled in CI/CD automation using GitHub Actions for platform and pipeline deployments.

Proven ability to troubleshoot, optimize, and support mission‑critical data workloads in insurance.

Strong documentation and communication skills to support cross‑functional teams and stakeholders.

**This position is offered through a strategic partnership between Don Nelson Technology Limited and NP Solution Limited.