Distributed ML Training Engineer - Large Models

Reflection AI

San Francisco (CA)

On-site

USD 180,000 - 300,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Benefits offered by this job

Top-tier compensation
Stock options
Health & wellness
Meals provided in office
Parental leave

Job summary

Reflection is a research lab focused on open, accessible AI, seeking an experienced engineer to build and scale distributed training systems for frontier model pre-training. You will work with researchers to operate large-scale runs across thousands of GPUs, optimizing throughput and efficiency while translating experiments into production-ready training pipelines.

The role emphasizes collaboration with ML researchers, debugging across GPU compute stacks, and improving distributed training

Qualifications

  • Experience building distributed training systems for large ML models.
  • Proficiency with Megatron, DeepSpeed or equivalent frameworks.
  • Expertise in GPU utilization and NCCL for high-performance training.
  • Strong debugging skills across distributed training stacks.
  • Ability to collaborate with ML researchers to productionize training workflows.

Responsibilities

  • Build and scale distributed training systems powering frontier model pre-training.
  • Operate large-scale training runs across thousands of GPUs.
  • Optimize training throughput, memory usage, and GPU utilization.
  • Develop scalable training pipelines with checkpointing and experiment management.
  • Debug performance bottlenecks across distributed stacks such as model parallelism and GPU communication.

Skills

Distributed training
Megatron/DeepSpeed
GPU optimization
NCCL
Debugging GPUs
Research collaboration

Tools

Megatron-LM
DeepSpeed

Job description

Reflection is a research lab focused on open, accessible AI, seeking an experienced engineer to build and scale distributed training systems for frontier model pre-training. You will work with researchers to operate large-scale runs across thousands of GPUs, optimizing throughput and efficiency while translating experiments into production-ready training pipelines.

The role emphasizes collaboration with ML researchers, debugging across GPU compute stacks, and improving distributed training

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Distributed Training Systems Engineer
Distributed Training Systems Engineer

Reflection AI Ltd • New York (NY)

On-site
USD 180,000 - 270,000
Top-tier compensation
Stock options
Health & wellness
+5
Distributed ML Training Infra Engineer
Distributed ML Training Infra Engineer

Thinking Machines Lab • San Francisco (CA)

On-site
USD 300,000 - 400,000
Health insurance
Dental insurance
Vision benefits
+3
Distributed ML Infra Engineer for Large-Scale Pre-Training
Distributed ML Infra Engineer for Large-Scale Pre-Training

Mohamed bin Zayed University of Artificial Intelligence • Sunnyvale (CA)

On-site
USD 150,000 - 450,000
Medical, dental, vision
Bonus
401K Plan
+4
Distributed ML Infra Engineer - Large-Scale Pre-Training
Distributed ML Infra Engineer - Large-Scale Pre-Training

Institute of Foundation Models • Sunnyvale (CA)

On-site
USD 150,000 - 450,000
Medical, dental, and vision benefits
Bonus
401K Plan
+4
Distributed Training Infra Engineer for Large Models
Distributed Training Infra Engineer for Large Models

Kindredventures • San Francisco (CA)

On-site
USD 180,000 - 240,000
Staff Engineer - Distributed AI Infrastructure & Systems
Staff Engineer - Distributed AI Infrastructure & Systems

Micro1 • United States

Remote
USD 180,000 - 260,000
Senior Research Engineer, Large-Scale ML Pre-training
Senior Research Engineer, Large-Scale ML Pre-training

Magic • San Francisco (CA)

On-site
USD 275,000 - 474,000
Equity in total compensation
401(k) with matching
Health, dental and vision insurance
+1
Senior ML Infra Research Engineer — Distributed Training
Senior ML Infra Research Engineer — Distributed Training

present of AI • San Francisco (CA)

On-site
USD 180,000 - 250,000
Senior ML Infra Architect – Distributed Training & GPUs
Senior ML Infra Architect – Distributed Training & GPUs

Xperteez Technology • United States

Remote
USD 180,000 - 260,000
Staff ML Engineer — Open Models & RL Research
Staff ML Engineer — Open Models & RL Research

Reflection AI Ltd • New York (NY)

On-site
USD 180,000 - 240,000
Top-tier compensation
Stock options
Health & wellness
+5