Data Platform Integration Engineer

Itmax System Berhad; Selangor; On-site; MYR 80,000 - 120,000; Full time

Job summary

A technology solutions provider in Malaysia is seeking a Data Platform Integration Engineer to design and deploy a modern on-premise data platform. This role involves integrating various systems, operating containers, and ensuring platform reliability. The ideal candidate has strong skills in middleware, Docker, Kubernetes, and Apache Kafka. Those with a fundamental understanding of streaming systems and willing to learn are encouraged to apply. This position offers hands-on experience and a clear growth path towards senior roles.

Benefits

Ownership of a production data platform

Hands-on experience with streaming systems

Clear growth path to Senior Platform Engineer

High-impact role in a focused engineering team

Qualifications

• Strong understanding of middleware and system integration.
• Hands-on experience with Docker and Kubernetes or Docker Swarm.
• Working knowledge of Apache Kafka.
• Experience operating stateful distributed systems.
• Experience with Flink and/or Spark.
• Understanding of stream vs batch, event-time & windowing, and stateful processing.
• Production experience with PostgreSQL.

Responsibilities

• Containerise, deploy, and operate Kafka, Flink, Spark, and Airflow.
• Manage on-prem container platforms (Kubernetes or Docker Swarm).
• Integrate PostgreSQL and other systems into Kafka.
• Support low-latency real-time processing and scheduled batch analytics.
• Implement monitoring and alerting using ELK or LGTM stack.
• Design for high availability and disaster recovery.

Skills

Middleware and system integration

Docker

Kubernetes

Apache Kafka

Stateful distributed systems

Flink

Spark

PostgreSQL

ELK

Grafana

Prometheus

Iceberg

Tools

Kafka

Flink

Spark

Kubernetes

Docker Swarm

PostgreSQL

Job description

We are hiring a Data Platform Integration Engineer to design, deploy, operate and monitor a modern on‑premise data platform supporting real‑time streaming, batch analytics, and map‑based live operational dashboards.

This is a hands‑on platform and integration role, working with Kafka, Flink, Spark, Kubernetes, and S3‑compatible data lakes, focused on reliability, scalability, and production operations.

Kubernetes or Docker Swarm, PostgreSQL

Key Responsibilities

Platform & Infrastructure

• Containerise, deploy, and operate:
  • Kafka, Flink, Spark, Airflow
  • S3‑compatible object storage (Ozone)
• Manage on‑prem container platforms (Kubernetes or Docker Swarm)
• Handle networking, storage, scaling, service discovery and onboarding of new data sources

Data & System Integration

• Integrate PostgreSQL and other systems into Kafka (CDC or polling)
• Build fault‑tolerant, ordered streaming pipelines
• Support low‑latency real‑time processing and scheduled batch analytics
• Manage schema evolution and data lifecycle using Iceberg

Observability & Operations

• Implement monitoring and alerting using ELK or LGTM stack
• Monitor cluster health, Kafka consumer lag, Flink / Spark job performance and storage utilisation
• Perform production troubleshooting and root cause analysis

Reliability & Security

• Design for high availability and disaster recovery
• Secure internal services, credentials, and network access
• Optimise resource usage for on‑prem constraints

Requirements

We do not expect every candidate to meet every requirement, but strong experience in most of the following is important.

Core Skills

• Strong understanding of middleware and system integration
• Hands‑on experience with Docker and Kubernetes or Docker Swarm
• Working knowledge of Apache Kafka
• Experience operating stateful distributed systems

Data, Streaming & Observability

• Experience with Flink and/or Spark
• Understanding of stream vs batch, event‑time & windowing and stateful processing
• Production experience with PostgreSQL
• Experience with ELK, Grafana, Prometheus
• Exposure to Iceberg, Hudi, or Delta Lake is a plus

Who Should Apply

• Engineers interested in platform engineering and system integration
• Backend or data engineers moving into distributed systems
• Strong fundamentals with willingness to learn are welcome

What We Offer

• Ownership of a production data platform
• Hands‑on experience with Kafka, Kubernetes, and streaming systems
• Clear growth path to Senior Platform Engineer / Solutions Architect
• High‑impact role in a focused engineering team