Staff Engineer: Scalable ML Training & Distributed Systems

Reflection; New York (NY); On-site; USD 180,000 - 260,000; Full time

Benefits offered by this job

Top-tier compensation

Stock options

Health & wellness

Meals provided

Parental leave

Unlimited PTO (US)

Visa sponsorship

Team events

Job summary

Reflection is seeking a software engineer to design and optimize the core training infrastructure powering our models. You will work on RL training loops, distributed GPU systems, and large-scale data pipelines, collaborating with researchers to translate ideas into reliable, scalable systems.

You will tackle bottlenecks, implement state-of-the-art techniques, and build tooling for launching, monitoring, and reproducing experiments, with a focus on performance, stability, and reproducibility.

Qualifications

• Strong software engineer who speaks the language of machine learning.
• Experience implementing research papers in production-ready systems.
• Deep experience in Distributed Training & Inference or Data Infrastructure.

Responsibilities

• Designing and optimizing large-scale training loops and data pipelines.
• Implementing state-of-the-art techniques with numerical stability and efficiency.
• Building internal tooling for launching, monitoring, and reproducing experiments.
• Diagnosing bottlenecks across the training stack (GPU memory, communication, dataloader).
• Translating research prototypes into production-grade infrastructure.

Skills

Distributed Training & Inference

Data Infrastructure

High-performance computing

Numerical stability

ML software engineering

Tools

Ray

Kubernetes

Slurm

Containerization

Sandboxing

Job description

Reflection is seeking a software engineer to design and optimize the core training infrastructure powering our models. You will work on RL training loops, distributed GPU systems, and large-scale data pipelines, collaborating with researchers to translate ideas into reliable, scalable systems.

You will tackle bottlenecks, implement state-of-the-art techniques, and build tooling for launching, monitoring, and reproducing experiments, with a focus on performance, stability, and reproducibility.