Distributed ML Training Systems Architect

Mind Robotics; Palo Alto (CA); On-site; USD 180,000 - 240,000; Full time

Job summary

Mind Robotics is seeking a Model Systems Engineer to build core systems that enable fast, reliable, and scalable model training—from experimentation to production deployment.

You will design and implement scalable training systems, optimize distributed workflows across hundreds of GPUs, and collaborate closely with modeling teams to accelerate iteration speed while reducing training costs. Expect a rigorous, infrastructure-focused role in a cutting-edge robotics lab.

Qualifications

• Experience building infrastructure for large-scale ML training.
• Deep understanding of how modern LLM/VLM systems are trained and scaled.
• Proven experience setting up and scaling distributed training across hundreds of GPUs.

Responsibilities

• Design and implement scalable systems for training large ML models.
• Enable efficient workflows for data ingestion, training, and iteration.
• Develop and optimize distributed training systems across hundreds of GPUs.
• Implement strategies for parallelization, sharding, and efficient compute utilization.
• Improve training efficiency through attention optimization, kernel fusion, and memory management.
• Partner with modeling teams to accelerate iteration speed and reduce training costs.
• Build internal tools for experiment tracking, monitoring, and debugging.
• Track training performance, failures, and resource utilization.
• Debug and resolve bottlenecks across the training stack.
• Provide lightweight infrastructure support for deploying and running models in production environments.
• Optimize inference performance and reliability where needed.
• Support core cloud infrastructure needs for training workloads.

Skills

Python

Distributed training

Memory management

Parallelism

Tools

PyTorch

JAX

Job description

Mind Robotics is seeking a Model Systems Engineer to build core systems that enable fast, reliable, and scalable model training—from experimentation to production deployment.

You will design and implement scalable training systems, optimize distributed workflows across hundreds of GPUs, and collaborate closely with modeling teams to accelerate iteration speed while reducing training costs. Expect a rigorous, infrastructure-focused role in a cutting-edge robotics lab.