Software Engineer (Big Data)

TikTok Pte. Ltd.; Singapore; On-site; SGD 60,000 - 90,000; Full time

Job summary

A leading technology firm in Singapore is seeking a skilled data engineer to design and implement real-time data systems for large-scale recommendation systems. The ideal candidate will have proficiency in Java, C++, Scala, and Python, along with strong troubleshooting skills and understanding of streaming computing systems. A passion for learning new technologies and working with massive datasets is essential. Experience with Flink and data lake technologies is preferred. This position offers a dynamic work environment and opportunities for growth.

Qualifications

• Proficient in programming languages like Java, C++, Scala, Python.
• Strong coding and troubleshooting skills.
• At least 1 year of relevant experience in software development.
• Deep understanding of streaming computing systems with production experience.

Responsibilities

• Design and implement real-time data systems for recommendation systems.
• Create high-performance storage systems and computing models.
• Troubleshoot production system failures and ensure stability.
• Research and develop computer software or utility programs.

Skills

Java

C++

Scala

Python

FlinkDataStream

FlinkSQL

data lake technology

troubleshooting

Job description

Responsibilities

Team Introduction TikTok's Recommendation Architecture Team is responsible for real-time computing direction, handling the design and development of real-time computing systems for TikTok videos, live streams, e-commerce, and a billion-user product recommendation system. Their main focus is ensuring system stability and high availability. They abstract general real-time computing systems, build a unified recommendation feature middleware, and implement a flexible and scalable high-performance storage system and computing model.

This enables advanced real-time data systems for deduplication, counting, feature services, and other recommendation-related business needs. Responsibilities:

• Design and implement real-time (streaming computing) data systems for large-scale recommendation systems.
• Create flexible, scalable, stable, and high-performance storage systems and computing models.
• Troubleshoot production system failures, design and implement necessary mechanisms and tools to ensure overall stability of the production systems
• Construct industry-leading streaming computing frameworks and other distributed systems to provide reliable infrastructure for massive data and large-scale business systems
• Research, design, and develop computer and network software or specialised utility programs.
• Analyse user needs and develop software solutions, applying principles and techniques of computer science, engineering, and mathematical analysis.
• Update software, enhances existing software capabilities, and develops and direct software testing and validation procedures.
• Work with computer hardware engineers to integrate hardware and software systems and develop specifications and performance requirements.

Qualifications

Minimum Qualifications

• Proficient in programming languages like Java, C++, Scala, Python.
• Strong coding and troubleshooting skills.
• At least 1 year of relevant experience
• Deep understanding of streaming computing systems, with formal production experience in developing TB-level Flink real-time computing systems.

Proficient in modules like FlinkDataStream, FlinkSQL, FlinkCheckpoint, FlinkState, and preferably with experience in reading Flink source code.

• Experience in data lake development is preferred.

Familiar with at least one data lake technology such as Hudi, Iceberg, DeltaLake, and preferably with experience in reading their source code.

• Willingness to tackle problems without clear answers, with a strong passion for learning new technologies.

Preferred Qualifications

• Experience in handling PB-level data is a plus.
• Familiarity with other big data systems is preferred, including YARN, K8S, Spark, SparkSQL, Kudu, and others.
• Experience in storage systems such as Hbase, Cassandra, RocksDB.