Big Data Development Engineer

CEKAP LINK TECH SDN. BHD.; Kuala Lumpur; On-site; MYR 180,000 - 320,000; Full time

Job summary

CEKAP LINK TECH SDN. BHD. is seeking an experienced big data developer to design and implement high-availability data architectures in a Kuala Lumpur-based role.

You will independently develop platform modules, optimize large-scale data processing with Spark and Flink, and collaborate with data analysts and algorithm teams to deliver efficient data interfaces. Requirements include a Bachelor's degree in CS and 3+ years in big data, with proven experience in 50TB+ data platforms and expertise in

Qualifications

• Bachelor's degree or above in computer science preferred.
• 3+ years of experience in big data development.
• Experience leading the construction of a data platform of 50TB or more or reengineering core modules.
• Proficient in Hadoop/Spark/Flink ecosystem architecture.
• Familiar with Kafka/Pulsar stream architectures.
• Proficient in SQL/Hive/Python/Java/Scala.
• Familiar with Doris/StarRocks and SQL tuning.
• Understanding of full data scenarios in at least one vertical (e.g., logistics, e-commerce).
• Able to propose technical paths for business growth.
• Data product thinking and ability to translate business requirements into implementable data solutions.

Responsibilities

• Design and implement high availability and high scalability big data architectures.
• Optimize performance of large-scale data processing and reduce task times.
• Develop data platform modules independently and resolve data processing challenges.
• Collaborate with analysts and algorithm teams to provide data interfaces and tools.
• Lead or participate in designing technical solutions and producing implementation documents.

Skills

Big data thinking

Business requirements

Education

Bachelor's degree in Computer Science

3+ years experience in big data development

Tools

Hadoop

Spark

Flink

Hive

Doris

StarRocks

Kafka

Pulsar

Python

Java

Scala

Job description

This is a big data development role focused on designing and implementing high availability and high scalability big data architectures. You will optimize performance of large-scale data processing systems, independently develop big data platform modules, and collaborate with data analysts and algorithm teams to provide efficient data interfaces and tool support.

Key responsibilities

Participate in high availability and high scalability big data architectures (such as the selection of Lambda/Kappa architectures), and implement technical solutions under guidance

Optimize the performance of large-scale data processing (such as reducing task execution time by over 30% through Spark tuning, and reducing memory overflow through Flink state backend optimization)

Independently complete the development of big data platform modules, and solve technical difficulties in the data processing chain

Collaborate with data analysts or algorithm teams to provide efficient data interfaces and tool support

Lead or participate in the design of technical solutions, output development documents, and promote implementation

About you

Bachelor's degree or above, with a major in computer science preferred

More than 3 years of experience in big data development

Have led the construction of a data platform of 50TB or more or the reengineering of core modules

Proficient in the Hadoop/Spark/Flink ecosystem architecture

Familiar with the stream architecture design of Kafka/Pulsar

Proficient in SQL/Hive/Python/Java/Scala

Proficient in Doris/StarRocks parameters and SQL tuning

Familiar with the full data scenario of at least one vertical field (such as logistics, e-commerce, communication)

Able to propose technical paths for business growth

Have a certain data product thinking, and able to abstract business requirements into implementable data solutions