Distributed Training Systems Engineer

Reflection AI Ltd; New York (NY); On-site; USD 180,000 - 270,000; Full time

Benefits offered by this job

Top-tier compensation

Stock options

Health & wellness

Meals provided

Parental leave

Unlimited PTO

Sponsorship visas

Team events

Job summary

Reflection AI Ltd in New York seeks a senior distributed training engineer to build and scale systems powering frontier ML model pre-training.

You will design and operate large-scale training runs across thousands of GPUs, developing infrastructure with Megatron, DeepSpeed, and NCCL, and optimizing throughput and memory usage.

Join a mission to make intelligence open and accessible, with stock options, competitive compensation, and visa sponsorship where applicable.

Qualifications

• Experience building or operating distributed training systems for large ML models.
• Experience with Megatron, DeepSpeed or similar large-scale training systems.
• Familiarity with large-scale model parallelism strategies (data, tensor, pipeline, or expert parallelism).
• Experience optimizing training throughput and GPU utilization in large distributed environments.
• Familiarity with GPU communication libraries such as NCCL and performance tuning for distributed workloads.
• Experience working closely with ML researchers to productionize experimental training workflows.
• Strong debugging skills across GPU compute, distributed training systems, and large-scale ML pipelines.
• Experience working with large datasets and training pipelines used for foundation model pre-training.

Responsibilities

• Build and scale distributed training systems for frontier models.
• Design and operate large-scale training runs across thousands of GPUs.
• Develop infrastructure for efficient training across thousands of GPUs using Megatron, DeepSpeed, NCCL.
• Optimize training throughput, stability, and performance of large workloads.
• Collaborate with researchers to productionize experimental training workflows.
• Improve GPU utilization and memory management in distributed stacks.
• Maintain training pipelines for large datasets, checkpoints, and iterations.
• Debug performance bottlenecks across distributed training stacks, including model parallelism and GPU communication.
• Contribute to rapid experimentation and iteration on new training techniques.

Skills

Distributed training

Megatron/DeepSpeed

Model parallelism

GPU utilization

Debugging pipelines

Collaboration with researchers

Tools

Megatron

DeepSpeed

NCCL

Job description

Reflection AI Ltd in New York seeks a senior distributed training engineer to build and scale systems powering frontier ML model pre-training.

You will design and operate large-scale training runs across thousands of GPUs, developing infrastructure with Megatron, DeepSpeed, and NCCL, and optimizing throughput and memory usage.

Join a mission to make intelligence open and accessible, with stock options, competitive compensation, and visa sponsorship where applicable.