LLM Pretraining Architect

Kuailu Software; Singapore; On-site; SGD 180,000 - 260,000; Full time

Job summary

Kuailu Software seeks an expert for leading large language model pre-training initiatives. You will design data strategies, supervise data engineering, and optimize distributed training across multi-GPU setups. Expertise with Megatron-LM, DeepSpeed, and FSDP is essential for scaling models >7B parameters.

The role focuses on long-context training techniques, monitoring training health, and iterating based on regular evaluations to improve model quality.

Qualifications

• Bachelor’s degree or higher in AI, NLP, CS, or a related field.
• Hands-on experience pre-training large language models (>=7B params).
• Strong background in distributed training frameworks (Megatron-LM, DeepSpeed, FSDP).
• Experience with data cleaning, deduplication, tokenization, data mixing.
• Familiarity with monitoring and debugging training processes.
• Experience with long-context training techniques.

Responsibilities

• Design pre-training strategies (data composition, LR schedule, batch size ramp-up, long sequence strategies).
• Lead pre-training data engineering (deduplication, PII filtering, quality classification, tokenization, data mixing, curriculum learning).
• Configure and optimize large-scale distributed training (3D parallelism, GPU memory, communication).
• Develop long-context training solutions (RoPE scaling, NTK interpolation, YaRN).
• Monitor training performance (loss analysis, gradient anomaly, stability, checkpointing).
• Perform intermediate evaluations and adjust data and strategies based on results.

Skills

LLM pre-training

Distributed training

Data engineering

Training monitoring

Long-context training

GPU memory optimization

Curriculum learning

PII filtering

Education

Bachelor’s degree or higher in AI, NLP, CS, or related field

Tools

Megatron-LM

DeepSpeed

FSDP

64 GPUs

3D parallelism

Job description

Kuailu Software seeks an expert for leading large language model pre-training initiatives. You will design data strategies, supervise data engineering, and optimize distributed training across multi-GPU setups. Expertise with Megatron-LM, DeepSpeed, and FSDP is essential for scaling models >7B parameters.

The role focuses on long-context training techniques, monitoring training health, and iterating based on regular evaluations to improve model quality.