Large Language Model Pre-training Engineer

UMELIFE (SINGAPORE) PTE. LTD.; Singapore; On-site; SGD 180,000 - 260,000; Full time

Job summary

UMELIFE (SINGAPORE) PTE. LTD. is seeking an expert in large-scale LLM pre-training to drive strategy, data pipelines, and distributed training across multi-GPU clusters.

The role requires hands-on experience with 7B+ parameter models, Megatron-LM, and DeepSpeed, with a focus on training stability and optimization. The candidate will work on RoPE scaling, NTK-aware interpolation, and YaRN-related techniques to extend long-context capabilities and improve efficiency in a high-performance

Qualifications

• Bachelor's degree or above in CS, AI, NLP, ML, distributed systems or related field.
• LLM pre-training experience and hands-on experience with complete LLM pre-training projects.
• Experience pre-training 7B+ parameter models.
• Experience with large-scale distributed training frameworks.
• 64+ GPU training environments experience.
• Solid understanding of large-scale pre-training data pipelines.
• Experience fine-tuning and optimization for long-context models.
• Publications in top AI/ML conferences are a plus.

Responsibilities

• Lead pre-training strategy and architecture for large-scale models.
• Handle pre-training data engineering and pipeline quality.
• Coordinate distributed training across 64+ GPUs.
• Monitor training, debug issues, and optimize convergence.
• Evaluate models and iteratively optimize training performance.
• Work on long-context training techniques and extensions.

Skills

LLM pre-training

Large-scale distributed training

Data engineering concepts

Training monitoring & debugging

Model evaluation & optimization

Troubleshooting distributed training

Long-context training

RoPE scaling awareness

NTK-aware interpolation

YaRN technique knowledge

Education

Bachelor's degree or above in Computer Science, AI, NLP, ML, Distributed Systems or related field

Tools

Megatron-LM

DeepSpeed

FSDP

RoPE scaling

NTK-aware interpolation

YaRN

Job description

JobResponsibilities

• Pre-trainingStrategy&Architecture
• Pre-trainingDataEngineering
• Large-scaleDistributedTraining
• Long-contextTraining
• TrainingMonitoring&Optimization
• Evaluation&IterativeOptimization

JobRequirements

• Bachelor'sdegreeoraboveinComputerScience,ArtificialIntelligence,NLP,MachineLearning,DistributedSystems,orarelatedfield.
• LLMPre-trainingExperience
• Hands-onexperiencewithcompleteLLMpre-trainingprojects.
• Experienceparticipatinginthepre-trainingof7B+parametermodels.
• DistributedTraining
• Strongexpertiseinlarge-scaledistributedtrainingframeworkssuchasMegatron-LM,DeepSpeed,orFSDP.
• Practicalexperiencewith64+GPUtrainingenvironments.
• Pre-trainingDataEngineering
• Solidunderstandingoflarge-scalepre-trainingdatapipelines,includingdatacleaning,deduplication,qualityfiltering,tokenization,datamixing,anddataqualityoptimization.
• TrainingMonitoring&Debugging
• Strongabilitytoanalyzetrainingloss,gradients,convergence,andtrainingstability.
• Experiencetroubleshootinglarge-scaledistributedtrainingissues.
• Long-contextTraining
• Familiaritywithlong-contexttrainingandextensiontechniques,includingRoPEscaling,NTK-awareinterpolation,andYaRN.
• PreferredQualifications
• Experiencewith70B+parametermodelpre-training.
• ExperiencewithMoE(Mixture-of-Experts)modelpre-training.
• Publicationsintop-tierAI/MLconferencessuchasNeurIPS,ICML,ICLR,ACL,orEMNLP,particularlyinLLMpre-training,modelarchitecture,ortrainingoptimization.
• Experienceoptimizinglarge-scaleGPUclustersandtraininginfrastructure.