Tech Lead: Scalable ML Systems & LLM Training

Scale; San Francisco, New York (CA, NY); On-site; USD 290,000 - 363,000; Full time

Benefits offered by this job

Health, dental & vision coverage

Retirement benefits

Learning stipend

Generous PTO

Commuter stipend

Job summary

Scale's LLM post-training platform team builds our internal distributed framework for large language model training. The platform powers MLEs, researchers, data scientists, and operators for fast and automatic training and evaluation of LLMs.

Scale is uniquely positioned at the heart of the field of AI as an indispensable provider of training and evaluation data and end-to-end solutions for the ML lifecycle.

Qualifications

• Passionate about system optimization.
• Experience with multi-node LLM training and inference.
• Experience with developing large-scale distributed ML systems.
• Experience with post-training methods like RLHF/RLVR and related algorithms like PPO/GRPO etc.
• Strong software engineering skills, proficient in frameworks and tools such as CUDA, Pytorch, transformers, flash attention, etc.
• Strong written and verbal communication skills to operate in a cross-functional team environment.

Responsibilities

• Build, profile and optimize our training and inference framework.
• Collaborate with ML and research teams to accelerate their research and development, and enable them to develop the next generation of models and data curation.
• Research and integrate state-of-the-art technologies to optimize our ML system.

Skills

Multi-node training

Distributed ML systems

RLHF/RLVR

PPO/GRPO

CUDA

PyTorch

Transformers

Flash Attention

Cross-functional comms

Tools

CUDA

PyTorch

Transformers

Flash Attention

Job description

Scale's LLM post-training platform team builds our internal distributed framework for large language model training. The platform powers MLEs, researchers, data scientists, and operators for fast and automatic training and evaluation of LLMs.

Scale is uniquely positioned at the heart of the field of AI as an indispensable provider of training and evaluation data and end-to-end solutions for the ML lifecycle.