Data Platform Integration Engineer

北京汉克时代科技有限公司; Kuala Lumpur; On-site; MYR 120,000 - 180,000; Full time

Job summary

北京汉克时代科技有限公司 is seeking an experienced Data Platform Engineer for the Kuala Lumpur team. You will maintain core data pipelines across NiFi, Kafka, Snowflake, and PostgreSQL, ensuring robust ingestion and real-time processing.

The role requires collaboration with overseas teams and a fully English working environment. You will handle incident response, capacity planning, and production deployments, using APM and log-monitoring tools to detect issues early and drive resolution.

Qualifications

• Bachelor's degree or above in Computer Science, Software Engineering, Information Management or related discipline.
• Around three years of experience in data platforms, data integration, or real-time data development.
• Hands-on Apache NiFi experience including ingestion, flow configuration and troubleshooting.
• Experience with Kafka or Azure Event Hub and understanding of messaging concepts.
• Experience with Snowflake data warehouse, loading and basic optimisation.
• Proficiency with PostgreSQL and database performance troubleshooting.
• Proficiency with Linux and using logs/CLI for issue location.
• Experience with monitoring tools like Splunk or Grafana.
• Familiar with Airflow and Azure Functions orchestration.
• Strong ownership, cross-team collaboration and ability to work under pressure.

Responsibilities

• Join the data platform operations team to maintain and optimise core integration pipelines.
• Maintain NiFi, Kafka and Snowflake ingestion pipelines for reliable data flows.
• Maintain Event Hub, Flink and PostgreSQL real-time processing pipelines.
• Manage daily operations, capacity assessment and incident handling for core tools.
• Collaborate with upstream/downstream teams on data onboarding and quality checks.
• Support production changes, releases, on-call response and incident handling.
• Use APM and log-monitoring tools to track pipeline health and drive issue resolution.
• Prepare English technical docs, operation manuals and handover materials.

Skills

NiFi

Kafka

Event Hub

Snowflake

PostgreSQL

Linux

Grafana

Airflow

Azure Functions

APM / logs

Cross-team collaboration

Education

Bachelor's degree or above in CS/SE/Info Mgmt

Tools

Splunk

Grafana

Airflow

Azure Functions

Job description

Join the data platform operations team to maintain and optimise core integration pipelines that support reliable data ingestion, real-time processing and data delivery. You will work across NiFi, Kafka, Snowflake, Azure Event Hub, Flink and PostgreSQL, collaborating with upstream and downstream teams to keep production data flows stable, observable and performant.

Key responsibilities

Perform daily maintenance, monitoring, troubleshooting and performance optimisation of core data platform integration pipelines.

Maintain NiFi, Kafka and Snowflake data ingestion and data-landing pipelines, ensuring reliable data flows from multiple source systems into Snowflake.

Maintain Event Hub, Flink and PostgreSQL real-time data processing pipelines, ensuring stable and reliable computation and writes.

Manage daily operations, capacity assessment and incident handling for Kafka, Event Hub, Flink, NiFi, Snowflake and PostgreSQL.

Work with upstream and downstream teams on data onboarding, interface integration, data quality checks and production issue diagnosis.

Support production environment changes, releases, on-call response and emergency incident handling.

Use APM and log-monitoring tools to track data pipeline health, detect anomalies early and drive issues through to resolution.

Prepare English technical documents, operation manuals, incident post-mortems and handover materials to support smooth project transitions.

Participate in a fully English working environment and communicate with overseas teams.

Support production on-call duties, change activities and emergency issue response, including short-term travel or cross-time-zone collaboration when required by the project.

What we're looking for

Bachelor's degree or above in Computer Science, Software Engineering, Information Management or a related discipline.

Around three years of experience in data platforms, data integration, big data operations or real-time data development.

Hands-on experience with Apache NiFi, including data ingestion, flow configuration, troubleshooting and performance optimisation.

Experience with Apache Kafka or Azure Event Hub and an understanding of messaging concepts, consumer lag handling and common operational issues.

Experience maintaining Apache Flink real-time stream processing jobs, including state management, checkpoints or troubleshooting.

Experience with Snowflake or another cloud data warehouse, including data loading, table design, basic query optimisation and access management.

Experience with PostgreSQL, including database connectivity, write performance, indexes, locks and common troubleshooting.

Proficiency with Linux, including using logs, monitoring and command-line tools to locate issues.

Ability to use monitoring tools such as Splunk and Grafana.

Ability to use orchestration tools such as Airflow and Azure Functions.

Strong ownership, problem-analysis and cross-team collaboration skills, with the ability to work effectively under pressure.

Experience with real-time data platforms, data pipelines, ETL/ELT or unified batch-and-stream projects is advantageous.

Experience with Azure, AWS or other cloud platform data services is advantageous.

Experience with APM, log monitoring, alerting framework build-out or production incident troubleshooting is advantageous.

Experience with international projects, overseas team collaboration or 24/7 production support is advantageous.