Azure Databricks Data Engineer - Pipelines, Quality & Governance

IT Channel (Asia) Limited; Hong Kong; On-site; HKD 480,000 - 720,000; Full time

Job summary

IT Channel (Asia) Limited in Hong Kong seeks a data engineer to design, build, and maintain scalable ETL/ELT pipelines on Azure Databricks for diverse insurance data, collaborating with data architects and stakeholders to deliver robust assets. You will ensure data quality, governance, and security while integrating batch and streaming sources such as Kafka.

Proficiency in PySpark/Scala, Python, and Informatica, plus CI/CD with GitHub Actions, is essential for timely analytics and regulatory

Qualifications

• Bachelor's degree in Information Technology, Computer Science, Data Engineering, or related discipline.
• 3+ years of experience as a data engineer, building and maintaining ETL/ELT processes on Azure Databricks using PySpark or Scala.
• Strong experience orchestrating data ingestion, transformation, and loading with Azure Data Factory and Data Lake Gen2.
• Advanced proficiency in Python and Spark for data engineering and feature engineering in Databricks.
• Experience integrating batch and streaming data sources via Kafka or Azure Event Hubs for real-time insurance use cases.
• Hands-on use of Informatica for data quality, lineage, and governance to support regulatory standards.
• Familiarity with automation and CI/CD of Databricks workflows using GitHub Actions.
• Understanding of data security, RBAC, Key Vault, encryption, and compliance in the insurance sector.

Responsibilities

• Design, build, and maintain scalable ETL/ELT pipelines in Azure Databricks for insurance data.
• Collaborate with data architects, modellers, analysts, and stakeholders to gather requirements.
• Develop, test, and optimize data transformation routines, including batch and streaming solutions.
• Implement data quality, validation, and cleansing procedures to ensure reliability.
• Integrate Informatica tools for governance, lineage, and metadata capture as required.
• Ensure robust data security and compliance with HK PDPO and related regulations.
• Automate deployment pipelines using GitHub Actions for auditable data workflows.
• Produce comprehensive technical documentation for pipelines and processes.

Skills

PySpark

Scala

Azure Databricks

Azure Data Factory

Data Lake Gen2

Kafka

Informatica

Python

Spark

RBAC & security

Data governance

Education

Bachelor's degree in IT/CS/Data Engineering

Tools

Informatica

GitHub Actions

Azure Data Factory

Azure Databricks

Job description

IT Channel (Asia) Limited in Hong Kong seeks a data engineer to design, build, and maintain scalable ETL/ELT pipelines on Azure Databricks for diverse insurance data, collaborating with data architects and stakeholders to deliver robust assets. You will ensure data quality, governance, and security while integrating batch and streaming sources such as Kafka.

Proficiency in PySpark/Scala, Python, and Informatica, plus CI/CD with GitHub Actions, is essential for timely analytics and regulatory