Distributed ML Training Systems Engineer

Mind Robotics; Palo Alto (CA); On-site; USD 180,000 - 240,000; Full time

Job summary

Mind Robotics in Palo Alto is building generalized physical AI and seeks a Research Engineer to design scalable training systems for large ML models, enabling rapid experimentation and production deployment.

You will optimize distributed training across hundreds of GPUs, implement data/model/pipeline parallelism, and build internal tools for tracking experiments, monitoring, and debugging to accelerate researcher productivity.

Qualifications

• Experience building infrastructure for large-scale ML training.
• Understanding of how modern LLM/VLM systems are trained and scaled.
• Experience setting up and scaling distributed training across hundreds of GPUs.
• Knowledge of data/model/pipeline parallelism.
• Proficient in Python programming.
• Expert in PyTorch and/or JAX.
• Memory usage optimization techniques.

Responsibilities

• Design scalable systems for training large ML models.
• Enable efficient workflows for data ingestion, training, and iteration.
• Develop and optimize distributed training systems across hundreds of GPUs.
• Implement strategies for parallelization, sharding, and efficient compute utilization.
• Improve training efficiency through techniques such as attention optimizations, kernel fusion, and memory management.
• Partner with modeling teams to accelerate iteration speed and reduce training costs.
• Build internal tools for experiment tracking, monitoring, and debugging.
• Implement systems for tracking training performance, failures, and resource utilization.
• Debug and resolve bottlenecks across the training stack.
• Provide lightweight infrastructure support for deploying and running models in production environments.
• Optimize inference performance and reliability where needed.
• Support core cloud infrastructure needs for training workloads (without heavy DevOps overhead).
• Manage compute resources efficiently across training jobs.

Skills

Python

PyTorch

JAX

Distributed training

GPUs

Parallelization

Data ingestion

Job description

Mind Robotics in Palo Alto is building generalized physical AI and seeks a Research Engineer to design scalable training systems for large ML models, enabling rapid experimentation and production deployment.

You will optimize distributed training across hundreds of GPUs, implement data/model/pipeline parallelism, and build internal tools for tracking experiments, monitoring, and debugging to accelerate researcher productivity.