Senior Data Engineer — End-to-End Lakehouse Platform

ONL Biz Solutions Sdn Bhd; Kuala Lumpur; On-site; MYR 180,000 - 240,000; Full time

Job summary

ONL Biz Solutions Sdn Bhd is seeking a senior data engineer to own end-to-end data pipelines and governance for a lakehouse platform. You will work with Iceberg tables on AWS, Spark/Flink processing, and a ClickHouse serving layer to enable fast analytics.

Ideal candidates have 6+ years in data engineering, strong SQL and data modeling, and hands-on experience with distributed compute. Excellent collaboration and communication are essential for cross-team work.

Qualifications

• 6+ years in data engineering or data-intensive backend roles.
• Production experience with a lakehouse/open table format.
• Strong distributed compute: Spark and/or Flink; performance tuning a plus.
• Expert SQL and dimensional data modeling; proficient with analytical stores.
• Solid AWS fundamentals: S3, IAM, VPC, RDS.
• Production reliability mindset: monitoring, alerts, on-call, root-cause analysis.
• Proficiency in Python, Java, or Scala for pipelines.
• Excellent communication and stakeholder-management skills.

Responsibilities

• Own and operate the streaming platform for reliability, correctness, and cost.
• Build and tune batch (Spark) and real-time (Flink) processing pipelines.
• Develop and optimize the serving layer (ClickHouse over Iceberg) for fast queries.
• Design data models, partitioning, and compaction to balance write and query costs.
• Establish data governance: catalog, access control, data lineage, retention, quality checks.
• Collaborate with analysts, product teams, and engineers for accessible data access.
• Stay current with lakehouse ecosystem and propose improvements.

Skills

SQL

Python

Java

Scala

Spark

Flink

ClickHouse

AWS

Data modeling

Data governance

Tools

Iceberg

Kafka Connect

Flink

Spark

ClickHouse

S3

Terraform

Kubernetes

Dagster

Airflow

Job description

ONL Biz Solutions Sdn Bhd is seeking a senior data engineer to own end-to-end data pipelines and governance for a lakehouse platform. You will work with Iceberg tables on AWS, Spark/Flink processing, and a ClickHouse serving layer to enable fast analytics.

Ideal candidates have 6+ years in data engineering, strong SQL and data modeling, and hands-on experience with distributed compute. Excellent collaboration and communication are essential for cross-team work.