Research Engineer, Scalable ML Training & Infra

Mind Robotics; Palo Alto (CA); On-site; USD 170,000 - 260,000; Full time

Job summary

Mind Robotics is building generalized physical AI for real-world industrial deployment. We are seeking a Research Engineer to design scalable systems for training large ML models and to optimize distributed training across hundreds of GPUs.

You will work with Python, PyTorch or JAX, and contribute to data workflows, model iteration, and production deployment while building internal tools for experiment tracking and performance monitoring.

Qualifications

• Experience building infrastructure for large-scale ML training.
• Proven experience scaling distributed training across hundreds of GPUs.
• Strong Python, PyTorch, or JAX proficiency.

Responsibilities

• Design and implement scalable systems for training large ML models.
• Enable efficient workflows for data ingestion, training, and iteration.
• Develop and optimize distributed training systems across hundreds of GPUs.
• Implement strategies for parallelization, sharding, and compute utilization.
• Improve training efficiency with attention optimizations, kernel fusion, memory management.
• Partner with modeling teams to speed iteration and reduce training costs.
• Build internal tools for experiment tracking, monitoring, and debugging.
• Track training performance, failures, and resource utilization.
• Debug and resolve bottlenecks across the training stack.
• Provide lightweight infra support for deploying models in production.
• Optimize inference performance and reliability as needed.
• Support core cloud infra needs for training workloads.
• Manage compute resources across training jobs.

Skills

Distributed training

Python

PyTorch

JAX

Large-scale ML infra

Parallelization

GPU cluster management

Performance optimization

Experiment tooling

Tools

PyTorch

JAX

CUDA

Job description

Mind Robotics is building generalized physical AI for real-world industrial deployment. We are seeking a Research Engineer to design scalable systems for training large ML models and to optimize distributed training across hundreds of GPUs.

You will work with Python, PyTorch or JAX, and contribute to data workflows, model iteration, and production deployment while building internal tools for experiment tracking and performance monitoring.