Data Engineer

Senior Citizen Home Safety Association 長者安居協會; Hong Kong; On-site; HKD 600,000 - 900,000; Full time

Job summary

Senior Citizen Home Safety Association 長者安居協會 in Hong Kong seeks a hands-on Data Platform Engineer to build and maintain a scalable in-house data platform (Lakehouse/Data Mesh) supporting both high-volume batch processing and real-time streaming.

You will identify open-source tools, develop robust data pipelines to harmonize data from multiple systems into a single source of truth, and collaborate with the AI team on pipelines for AI applications and LLMOps.

Qualifications

• 3+ years of hands-on experience developing and deploying data platform solutions with modern data technologies.
• Experience with Data Warehousing (Snowflake, BigQuery, Redshift) and NoSQL (DynamoDB, MongoDB, Elasticsearch), Data Lakes (Databricks, Delta Lake).
• Proficiency in Python, SQL, and Java.
• Experience with cloud data services.

Responsibilities

• Set up, operate, and maintain a scalable in-house data platform for high-volume batch processing and real-time streaming.
• Identify and deploy open-source tools to enhance data management and analytics capabilities.
• Develop robust data pipelines to harmonize data from multiple systems to create a single source of truth; include complex transformations, deduplication and entity resolution.
• Collaborate with the AI team to set up data pipelines for AI applications and LLMOps.
• Ensure data is structured to support generative AI use cases and implement data governance to meet regulatory requirements.

Skills

Python

SQL

Java

Data Pipelines

Cloud Data Services

Tools

Snowflake

BigQuery

Redshift

DynamoDB

MongoDB

Elasticsearch

Databricks

Delta Lake

Job description

Set up, operate, and maintain a scalable in-house data platform (Lakehouse/Data Mesh) that supports both high-volume batch processing and real-time streaming.

Identify and deploy appropriate open-source tools to enhance data management and analytics capabilities.

Develop robust data pipelines to harmonize data from multiple disparate systems, performing complex transformations, deduplication, and entity resolution to create a "single source of truth."

Collaborate with the AI team to set up data pipelines specifically for AI applications and LLMOps.

Ensure data is structured to support generative AI use cases.

Implement and maintain data governance frameworks to ensure compliance with relevant regulatory requirements.

Requirements:

3+ years of hands-on experience developing and deploying solutions with modern data technologies, including but not limited to Data Warehousing (Snowflake, BigQuery, Redshift), NoSQL (DynamoDB, MongoDB, Elasticsearch), Data Lakes (Databricks, Delta Lake), or Customer Data Platform

Proficiency in Python, SQL, and Java

Experience with cloud data services