ML Platform Engineer: Training Orchestration & Observability

Rhoda AI; Mountain View (WY); On-site; USD 150,000 - 230,000; Full time

Job summary

Rhoda AI is hiring a Research Engineer to build and maintain a scalable training platform that powers model development. You will orchestrate large-scale GPU-based training, manage experiments, and develop observability to drive rapid iteration from idea to results.

You’ll collaborate with teams across training systems, data infrastructure, and research, ensuring reliable, reproducible workflows and efficient resource utilization for cutting-edge research in generalist robotics.

Qualifications

• Experience in building and operating ML platforms for large-scale training.
• Strong background in software engineering with focus on reliability and observability.
• Familiarity with distributed training frameworks and GPU orchestration.

Responsibilities

• Build and maintain training orchestration systems for large-scale distributed model training across GPU clusters
• Develop experiment management tooling: job configuration, tracking, reproducibility, and artifact management
• Build observability infrastructure for training runs: loss curves, compute utilization, gradient statistics, and anomaly detection
• Optimize and automate the research iteration loop from experiment launch to results analysis
• Manage job scheduling and cluster utilization for efficient use of GPU compute
• Build internal tooling and interfaces that help researchers move faster
• Collaborate with training systems, data infrastructure, and research teams to support their platform needs

Skills

MLOps

Distributed training

Experiment tracking

GPU cluster management

Reliability engineering

Tools

PyTorch

Slurm

Kubernetes

MLflow

Weights & Biases

Job description

Rhoda AI is hiring a Research Engineer to build and maintain a scalable training platform that powers model development. You will orchestrate large-scale GPU-based training, manage experiments, and develop observability to drive rapid iteration from idea to results.

You’ll collaborate with teams across training systems, data infrastructure, and research, ensuring reliable, reproducible workflows and efficient resource utilization for cutting-edge research in generalist robotics.