AI Engineer

Kuailu Software; Singapore; On-site; SGD 180,000 - 260,000; Full time

Job summary

Kuailu Software seeks an expert for leading large language model pre-training initiatives. You will design data strategies, supervise data engineering, and optimize distributed training across multi-GPU setups. Expertise with Megatron-LM, DeepSpeed, and FSDP is essential for scaling models >7B parameters.

The role focuses on long-context training techniques, monitoring training health, and iterating based on regular evaluations to improve model quality.

Qualifications

• Bachelor’s degree or higher in AI, NLP, CS, or a related field.
• Hands-on experience pre-training large language models (>=7B params).
• Strong background in distributed training frameworks (Megatron-LM, DeepSpeed, FSDP).
• Experience with data cleaning, deduplication, tokenization, data mixing.
• Familiarity with monitoring and debugging training processes.
• Experience with long-context training techniques.

Responsibilities

• Design pre-training strategies (data composition, LR schedule, batch size ramp-up, long sequence strategies).
• Lead pre-training data engineering (deduplication, PII filtering, quality classification, tokenization, data mixing, curriculum learning).
• Configure and optimize large-scale distributed training (3D parallelism, GPU memory, communication).
• Develop long-context training solutions (RoPE scaling, NTK interpolation, YaRN).
• Monitor training performance (loss analysis, gradient anomaly, stability, checkpointing).
• Perform intermediate evaluations and adjust data and strategies based on results.

Skills

LLM pre-training

Distributed training

Data engineering

Training monitoring

Long-context training

GPU memory optimization

Curriculum learning

PII filtering

Education

Bachelor’s degree or higher in AI, NLP, CS, or related field

Tools

Megatron-LM

DeepSpeed

FSDP

64 GPUs

3D parallelism

Job description

Responsibilities

• Design pre-training technical strategies, including data composition, training hyperparameters (LR schedule, batch size ramp-up, progressive sequence length strategies), etc.
• Lead pre-training data engineering, including data cleaning (deduplication, PII filtering, quality classification), tokenization, data mixing strategies, and curriculum learning.
• Configure and optimize large-scale distributed training, including 3D parallelism, GPU memory optimization, and communication optimization.
• Develop and implement long-context training solutions, including RoPE scaling, NTK-aware interpolation, and YaRN.
• Monitor and optimize training performance, including loss curve analysis, gradient anomaly detection, training stability, and checkpoint management.
• Conduct regular intermediate evaluations and adjust data composition and training strategies based on evaluation results.

Responsibilities

• Bachelor’s degree or above in AI, NLP, Computer Science, Systems, or a related field.
• Proven end-to-end experience in large language model (LLM) pre-training projects, with hands-on experience pre-training models of 7B+ parameters.
• Strong expertise in large-scale distributed training frameworks, such as Megatron-LM, DeepSpeed, or FSDP, with hands-on experience training on 64+ GPUs.
• Strong understanding of pre-training data engineering, including data cleaning, deduplication, quality filtering, tokenization, and data mixture optimization.
• Familiarity with training monitoring and debugging, including loss analysis, gradient monitoring, and training stability management.
• Familiarity with long-context training techniques, including RoPE scaling, NTK-aware interpolation, and YaRN.