Big Data Development Enginee

CEKAP LINK TECH SDN. BHD.; Kuala Lumpur; On-site; MYR 120,000 - 180,000; Full time

Job summary

CEKAP LINK TECH SDN. BHD. in Kuala Lumpur seeks a senior big data engineer to design, develop and implement scalable data platforms. You will work with Lambda and Kappa architectures, optimize Spark and Flink, and ensure reliable data pipelines while collaborating with data analysts and engineers.

The role requires strong programming skills in SQL, Python/Java/Scala and experience with Hive and Doris/StarRocks.

Qualifications

• Bachelor's degree or higher in CS, SWE, IT or related fields.
• 3+ years of hands-on big data development experience.
• Experience with 50TB+ data volumes preferred.
• Strong knowledge of Hadoop ecosystem (Spark, Flink).
• Hands-on with streaming tech such as Kafka or Pulsar.
• Proficient in SQL, Hive and Python/Java/Scala.
• Familiar with Doris or StarRocks and performance tuning.

Responsibilities

• Design, develop and implement scalable big data platforms.
• Work with Lambda and Kappa architectures.
• Optimize Spark performance, Flink, memory management.
• Develop core modules and troubleshoot data pipelines.
• Collaborate with analysts and engineers to provide data interfaces.
• Prepare design and development documentation.

Skills

Big data development

Spark

Flink

Kafka/Pulsar

SQL

Python/Java/Scala

Hive

Doris/StarRocks

Education

Bachelor's degree or above in Computer Science, Software Engineering, Information Technology or related disciplines

Tools

Hadoop ecosystem

Hive

Doris/StarRocks

Kafka

Pulsar

Iceberg/Delta Lake

Kubernetes

EMR/CDH

Job description

Participate in the design, development and implementation of highly available and scalable big data platforms and data processing solutions.

Work with big data architecture frameworks such as Lambda and Kappa, and support the implementation and optimisation of technical solutions.

Optimise large-scale data processing workloads, including Spark performance tuning, Flink optimisation, memory management, data processing efficiency and system stability improvements.

Develop, maintain and enhance core modules of the big data platform, and troubleshoot technical issues across data processing pipelines.

Collaborate with data analysts, algorithm engineers and other technical teams to provide reliable data interfaces, data services and supporting tools.

Participate in technical solution design, evaluate implementation approaches, and ensure solutions are scalable, stable and maintainable.

Prepare and maintain technical design documents, development documentation and implementation records to support successful project delivery.

Requirements

Bachelor's degree or above in Computer Science, Software Engineering, Information Technology or related disciplines.

3+ years of hands-on experience in big data development, with experience in building, implementing or optimising large-scale data platforms.

Experience working with large-scale data processing scenarios, including data platforms or core module development involving 50TB+ data volumes is preferred.

Strong knowledge of the Hadoop ecosystem, including Spark and Flink architecture, development and performance optimisation.

Hands-on experience with streaming data technologies such as Kafka or Pulsar.

Strong programming and data processing skills with SQL, Hive and at least one programming language such as Python, Java or Scala.

Experience with analytical database technologies such as Doris or StarRocks, including configuration, SQL optimisation and performance tuning.

Experience in distributed system optimisation, such as Spark Shuffle optimisation, Flink backpressure handling, state management or resource optimisation is preferred.

Experience with data lake technologies such as Apache Iceberg, Delta Lake or Apache Paimon is a plus.

Experience with cloud-native big data environments, including EMR, CDH or Kubernetes-based resource scheduling and cluster operations is a plus.

Experience in logistics, e-commerce, telecommunications or other data-intensive industries is preferred.

Strong analytical thinking, problem-solving skills and ability to collaborate effectively with cross-functional teams.

Strong communication skills with the ability to work effectively in a multilingual and multicultural environment.