LLM Pre-Training Engineer

KUAILU SOFTWARE (SINGAPORE) PTE. LTD.; Singapore; On-site; SGD 120,000 - 180,000; Full time

Job summary

KUAILU SOFTWARE (SINGAPORE) PTE. LTD. is seeking an expert in large-scale pre-training to design and optimize strategies for training cutting-edge language models.

You will lead data engineering, configure distributed GPU training, and develop long-context training techniques, with responsibilities spanning data cleaning, tokenization, and performance monitoring. The role requires hands-on experience with Megatron-LM, DeepSpeed, or FSDP on multi-GPU clusters, and a strong background in RoPE

Qualifications

• Bachelor’s degree or above in AI, NLP, Computer Science, Systems, or a related field.
• Proven end-to-end experience in large language model (LLM) pre-training projects, with hands-on experience pre-training models of 7B+ parameters.
• Strong expertise in large-scale distributed training frameworks, such as Megatron-LM, DeepSpeed, or FSDP, with hands-on experience training on 64+ GPUs.
• Strong understanding of pre-training data engineering, including data cleaning, deduplication, quality filtering, tokenization, and data mixture optimization.
• Familiarity with training monitoring and debugging, including loss analysis, gradient monitoring, and training stability management.
• Familiarity with long-context training techniques, including RoPE scaling, NTK-aware interpolation, and YaRN.

Responsibilities

• Design pre-training technical strategies, including data composition and training hyperparameters.
• Lead pre-training data engineering, including data cleaning, tokenization, and data mixing strategies.
• Configure and optimize large-scale distributed training, including 3D parallelism and GPU memory optimization.
• Develop and implement long-context training solutions, including RoPE scaling, NTK-aware interpolation, and YaRN.
• Monitor and optimize training performance, including loss curves and checkpoint management.
• Conduct regular evaluations and adjust data composition and training strategies based on results.

Skills

LLM pre-training

Distributed training

Data engineering

Model scaling

Training monitoring

RoPE scaling

NTK-aware interpolation

YaRN

Curriculum learning

Data cleaning

Education

Bachelor’s degree or above in AI/NLP/CS

Tools

Megatron-LM

DeepSpeed

FSDP

64+ GPUs

Job description

KUAILU SOFTWARE (SINGAPORE) PTE. LTD. is seeking an expert in large-scale pre-training to design and optimize strategies for training cutting-edge language models.

You will lead data engineering, configure distributed GPU training, and develop long-context training techniques, with responsibilities spanning data cleaning, tokenization, and performance monitoring. The role requires hands-on experience with Megatron-LM, DeepSpeed, or FSDP on multi-GPU clusters, and a strong background in RoPE