Senior ML Training Systems Engineer: Scale Multimodal AI

Roboticsconnected; Mountain View (WY); On-site; USD 180,000 - 260,000; Full time

Job summary

Rhoda AI is seeking a Staff / Principal ML Training Systems Engineer to own training systems performance end-to-end. You will define how models train at scale, driving efficiency, scalability, and correctness across large-scale multimodal training.

This is a core systems role focused on compute, GPU scaling, and rapid research iteration. Responsibilities include diagnosing bottlenecks in distributed training, crafting parallelism strategies, and extending training frameworks where needed.

Qualifications

• Proven track record improving large-scale distributed training performance.

Responsibilities

• Own training performance end-to-end.
• Diagnose and improve performance of large-scale multimodal training (vision, video, proprioception, actions, language).
• Build systematic performance attribution: step-time decomposition, scaling curves across cluster sizes, bottleneck identification and prioritization.
• Drive measurable gains in distributed, compute, and memory efficiency.
• Design and evolve parallelism strategies: data/tensor/pipeline/sharding/hybrid approaches.
• Improve execution efficiency through communication scheduling and overlap, graph capture and runtime improvements.
• Contribute to and extend training frameworks where needed.
• Build tools to identify bottlenecks quickly and track performance across model families.
• Develop regression detection with microbenchmarks and performance baselines.
• Partner with researchers and work side-by-side with researchers and engineers.

Skills

Distributed training

PyTorch

JAX

Data parallelism

Tensor parallelism

Pipeline parallelism

Sharded training

ZeRO

Systems thinking

Debugging

Profiling

Ownership

Tools

PyTorch

JAX

Job description

Rhoda AI is seeking a Staff / Principal ML Training Systems Engineer to own training systems performance end-to-end. You will define how models train at scale, driving efficiency, scalability, and correctness across large-scale multimodal training.

This is a core systems role focused on compute, GPU scaling, and rapid research iteration. Responsibilities include diagnosing bottlenecks in distributed training, crafting parallelism strategies, and extending training frameworks where needed.