Senior Distributed ML Training Engineer

Visa Hunt; San Francisco, Northern (CA, KY); Hybrid; USD 220,000 - 310,000; Full time

Benefits offered by this job

Stock options

Health/dental/vision insurance

Meals provided in office

Parental leave (22+ weeks)

Job summary

Reflection is seeking a role to build and operate distributed training systems powering frontier models in SF. You will work with research teams to design scalable training runs across thousands of GPUs, optimizing throughput and stability while advancing production-ready training pipelines.

This role emphasizes collaboration with ML researchers, debugging across GPU stacks, and improving communication and memory efficiency in large-scale training environments.

Qualifications

• Experience building or operating distributed training systems for large ML models.
• Strong experience with Megatron, DeepSpeed, or similar large-scale training frameworks.
• Familiarity with large-scale model parallelism strategies (data, tensor, pipeline, or expert).
• Experience optimizing training throughput and GPU utilization in large environments.
• Proven debugging skills across GPU compute and distributed training stacks.

Responsibilities

• Build and scale distributed training systems for frontier model pre-training.
• Operate large-scale training runs with research teams.
• Develop infrastructure for training across thousands of GPUs.
• Optimize throughput, stability, and efficiency of training workloads.
• Collaborate with researchers to productionize training workflows.
• Improve GPU communication and memory usage in distributed stacks.
• Maintain training pipelines with large datasets and checkpoints.
• Debug and resolve performance bottlenecks in model parallelism and runtime systems.

Skills

Distributed training systems

Megatron

DeepSpeed

Model parallelism

GPU optimization

NCCL

Debugging distributed stacks

Tools

Megatron

DeepSpeed

NCCL

Job description

Reflection is seeking a role to build and operate distributed training systems powering frontier models in SF. You will work with research teams to design scalable training runs across thousands of GPUs, optimizing throughput and stability while advancing production-ready training pipelines.

This role emphasizes collaboration with ML researchers, debugging across GPU stacks, and improving communication and memory efficiency in large-scale training environments.