Distributed ML Training Systems Engineer

Mind Robotics; Palo Alto (CA); On-site; USD 180,000 - 240,000; Full time

Job summary

Mind Robotics is hiring a Model Systems Engineer in Palo Alto to build core systems enabling fast, reliable training of large ML models and scalable production deployment.

You will design distributed training stacks, optimize memory and compute usage, and partner with modeling teams to accelerate iteration while reducing costs. Strong Python, PyTorch (and/or JAX), and memory optimization skills are essential.

Qualifications

• Experience building infrastructure for large-scale ML training.
• Deep understanding of how modern LLM/VLM systems are trained and scaled.
• Proven experience setting up and scaling distributed training across hundreds of GPUs.
• Strong understanding of parallelization strategies (data, model, pipeline parallelism).
• Proficiency in Python programming.
• Expert-level proficiency in PyTorch and/or JAX.
• Strong understanding of techniques like attention optimization, kernel fusion, and efficient memory usage.

Responsibilities

• Design and implement scalable systems for training large ML models.
• Enable efficient workflows for data ingestion, training, and iteration.
• Develop and optimize distributed training systems across hundreds of GPUs.
• Implement strategies for parallelization, sharding, and efficient compute utilization.
• Improve training efficiency through techniques such as attention optimizations, kernel fusion, and memory management.
• Partner closely with modeling teams to accelerate iteration speed and reduce training costs.
• Build internal tools for experiment tracking, monitoring, and debugging.
• Implement systems for tracking training performance, failures, and resource utilization.
• Debug and resolve bottlenecks across the training stack.
• Provide lightweight infrastructure support for deploying and running models in production environments.
• Optimize inference performance and reliability where needed.
• Support core cloud infrastructure needs for training workloads (without heavy DevOps overhead).
• Manage compute resources efficiently across training jobs.

Skills

Large-scale ML training

Distributed GPU training

Python programming

PyTorch

JAX

Parallelization strategies

Memory optimization

Experiment tooling

Tools

PyTorch

JAX

Job description

Mind Robotics is hiring a Model Systems Engineer in Palo Alto to build core systems enabling fast, reliable training of large ML models and scalable production deployment.

You will design distributed training stacks, optimize memory and compute usage, and partner with modeling teams to accelerate iteration while reducing costs. Strong Python, PyTorch (and/or JAX), and memory optimization skills are essential.