Data Engineer: Real-Time Big Data Pipelines

AI Rudder; Kuala Lumpur; On-site; MYR 90,000 - 130,000; Full time

Benefits offered by this job

AI/LLM projects

Growth potential

Competitive pay

Job summary

AI Rudder is building a data-heavy platform that uses AI voice automation to boost call center efficiency. The role focuses on designing scalable data pipelines and systems in a cross-functional team spanning China and Southeast Asia.

You will work with Kafka, HBase, Spark, and Flink, collaborating with data scientists and engineers to deliver robust data solutions and maintain system reliability.

Qualifications

• Experience with Kafka, HBase and Spark in production environments.
• Proficient in Java or Scala with 1+ year of development; Golang is a plus.
• Fluent English for technical discussions and collaboration.
• Understanding of distributed systems concepts: data consistency, fault tolerance, load balancing.

Responsibilities

• Design, develop, and optimize high-performance data pipelines and processing workflows (Kafka, HBase, Spark, Flink).
• Handle massive volumes of structured and unstructured data for real-time and batch processing.
• Monitor, tune, troubleshoot, and operate big data clusters for reliability and stability.
• Collaborate with data scientists, analysts, product managers, and engineers to translate requirements into robust solutions.
• Produce clear technical documentation and follow data governance and coding best practices.

Skills

Kafka

HBase

Spark

Java

Scala

Golang

Distributed systems

Fluent English

Mandarin

Tools

Flink

Airflow

Kubernetes

Hive

Iceberg

Delta Lake

Job description

AI Rudder is building a data-heavy platform that uses AI voice automation to boost call center efficiency. The role focuses on designing scalable data pipelines and systems in a cross-functional team spanning China and Southeast Asia.

You will work with Kafka, HBase, Spark, and Flink, collaborating with data scientists and engineers to deliver robust data solutions and maintain system reliability.