Staff Engineer: Distributed ML Training Systems

reflectionai; San Francisco, New York (CA, NY); On-site; USD 180,000 - 280,000; Full time

Benefits offered by this job

Stock options

Health insurance

Meals provided

Parental leave

Unlimited vacation

Visa sponsorship

Team off-sites

Job summary

reflectionai seeks a senior engineer to build and scale distributed training systems powering frontier model pre-training in San Francisco. You will partner with research teams to run large-scale foundation model training across thousands of GPUs, designing infrastructure for efficient, production-ready workflows.

You will optimize throughput, memory, and GPU utilization while debugging complex distributed pipelines and collaborating with researchers to push novel training techniques into

Qualifications

• Experience building or operating distributed training systems for large ML models.
• Strong experience with Megatron, DeepSpeed, or similar frameworks.
• Familiarity with model parallelism strategies (data/tensor/pipeline/expert).
• Experience optimizing throughput and GPU utilization in large distributed envs.
• Familiarity with NCCL and performance tuning for distributed workloads.
• Experience collaborating with ML researchers to productionize training workflows.
• Strong debugging across GPU compute, distributed systems, and ML pipelines.
• Experience handling large datasets and training pipelines for foundation models.

Responsibilities

• Build and scale distributed training systems that power frontier model pre-training.
• Work with research teams to run large-scale training for foundation models.
• Develop infra enabling training across thousands of GPUs using modern distributed frameworks.
• Optimize training throughput, stability, and efficiency for large workloads.
• Collaborate with researchers to translate ideas into production-ready training systems.
• Improve performance via optimization of communication, memory, and GPU utilization.
• Build and maintain training pipelines for large datasets, checkpoints, and iterations.
• Debug and resolve bottlenecks across distributed training stacks and model parallelism.

Skills

Distributed training

Megatron/DeepSpeed

GPU utilization

NCCL

Research collaboration

Performance profiling

Large-scale pipelines

Tools

Megatron

DeepSpeed

NCCL

Job description

reflectionai seeks a senior engineer to build and scale distributed training systems powering frontier model pre-training in San Francisco. You will partner with research teams to run large-scale foundation model training across thousands of GPUs, designing infrastructure for efficient, production-ready workflows.

You will optimize throughput, memory, and GPU utilization while debugging complex distributed pipelines and collaborating with researchers to push novel training techniques into