Databricks Data Engineer — Insurance Analytics on Azure

Don Nelson Recruitment Limited; Hong Kong; On-site; HKD 540,000 - 740,000; Full time

Job summary

Don Nelson Technology Limited seeks a skilled Data Engineer to design and maintain scalable Azure Databricks pipelines for insurance data. You will collaborate with architects, modellers, analysts and stakeholders to deliver robust data assets for analytics, operations, and regulatory reporting.

You will develop batch and streaming solutions using Azure Data Factory, Data Lake Gen2, Event Hubs and Kafka, with strong governance, security, and CI/CD practices.

Qualifications

• Bachelor's degree in CS/IT or related field.
• 3+ years with Azure cloud platforms and Databricks for insurance workloads.
• Experience provisioning and managing Databricks clusters/workspaces.
• Experience integrating Data Factory and Data Lake Gen2 with Databricks.
• Hands-on with Informatica for governance and metadata management.
• Strong Azure security, monitoring and cost optimization in regulated environments.
• CI/CD automation using GitHub Actions for deployment.
• Proven ability to troubleshoot data workloads in insurance.

Responsibilities

• Design and maintain scalable ETL/ELT pipelines in Azure Databricks.
• Collaborate with architects, modellers, analysts and stakeholders.
• Develop and optimize batch and streaming data solutions using ADF, Databricks, Kafka.
• Implement data validation and quality controls for critical use cases.
• Enforce governance, metadata management and cataloguing with Informatica.
• Apply domain expertise in Hong Kong Life and General Insurance.
• Automate deployment workflows with GitHub Actions.
• Troubleshoot pipelines and maintain detailed technical documentation.

Skills

Azure Databricks

Data Integration

Data Governance

Security & Compliance

CI/CD

Cloud Architecture

Education

Bachelor's degree in CS/IT

Tools

Azure Data Factory

Azure Data Lake Storage Gen2

Azure Event Hubs

Kafka

Informatica

Terraform

GitHub Actions

Job description

Don Nelson Technology Limited seeks a skilled Data Engineer to design and maintain scalable Azure Databricks pipelines for insurance data. You will collaborate with architects, modellers, analysts and stakeholders to deliver robust data assets for analytics, operations, and regulatory reporting.

You will develop batch and streaming solutions using Azure Data Factory, Data Lake Gen2, Event Hubs and Kafka, with strong governance, security, and CI/CD practices.