Research Engineer

Mind Robotics; Palo Alto (CA); On-site; USD 170,000 - 260,000; Full time

Job summary

Mind Robotics is building generalized physical AI for real-world industrial deployment. We are seeking a Research Engineer to design scalable systems for training large ML models and to optimize distributed training across hundreds of GPUs.

You will work with Python, PyTorch or JAX, and contribute to data workflows, model iteration, and production deployment while building internal tools for experiment tracking and performance monitoring.

Qualifications

• Experience building infrastructure for large-scale ML training.
• Proven experience scaling distributed training across hundreds of GPUs.
• Strong Python, PyTorch, or JAX proficiency.

Responsibilities

• Design and implement scalable systems for training large ML models.
• Enable efficient workflows for data ingestion, training, and iteration.
• Develop and optimize distributed training systems across hundreds of GPUs.
• Implement strategies for parallelization, sharding, and compute utilization.
• Improve training efficiency with attention optimizations, kernel fusion, memory management.
• Partner with modeling teams to speed iteration and reduce training costs.
• Build internal tools for experiment tracking, monitoring, and debugging.
• Track training performance, failures, and resource utilization.
• Debug and resolve bottlenecks across the training stack.
• Provide lightweight infra support for deploying models in production.
• Optimize inference performance and reliability as needed.
• Support core cloud infra needs for training workloads.
• Manage compute resources across training jobs.

Skills

Distributed training

Python

PyTorch

JAX

Large-scale ML infra

Parallelization

GPU cluster management

Performance optimization

Experiment tooling

Tools

PyTorch

JAX

CUDA

Job description

About Mind

Mind Robotics is building Physical AI for real-world industrial deployment, starting with the factory floor. We believe the hardest problems in AI are solved when researchers and engineers are hands-on with the physical world every day - and we're looking for people who are passionate about robotics, value ownership, and are excited to tackle difficult problems. Join us if you want to move beyond digital intelligence and put intelligence into motion.

About the team and the role

At Mind Robotics, we’re building generalized physical AI—robotic systems capable of dexterous, adaptive, and reasoning-intensive work in real-world industrial environments. Our ability to iterate quickly on large-scale models depends on world-class ML infrastructure. We’re looking for a Research Engineer to build the core systems that enable fast, reliable, and scalable model training—powering everything from experimentation to production deployment.

Responsibilities

• Design and implement scalable systems for training large ML models.
• Enable efficient workflows for data ingestion, training, and iteration.
• Develop and optimize distributed training systems across hundreds of GPUs.
• Implement strategies for parallelization, sharding, and efficient compute utilization.
• Improve training efficiency through techniques such as attention optimizations, kernel fusion, and memory management.
• Partner closely with modeling teams to accelerate iteration speed and reduce training costs.
• Build internal tools for experiment tracking, monitoring, and debugging.
• Implement systems for tracking training performance, failures, and resource utilization.
• Debug and resolve bottlenecks across the training stack.
• Provide lightweight infrastructure support for deploying and running models in production environments.
• Optimize inference performance and reliability where needed.
• Support core cloud infrastructure needs for training workloads (without heavy DevOps overhead).
• Manage compute resources efficiently across training jobs.

Requirements

• Strong experience building infrastructure for large-scale ML training.
• Deep understanding of how modern LLM/VLM systems are trained and scaled.
• Proven experience setting up and scaling distributed training across hundreds of GPUs.
• Strong understanding of parallelization strategies (data, model, pipeline parallelism).
• Strong proficiency in Python programming.
• Expert-level proficiency in PyTorch and/or JAX.
• Strong understanding of techniques like attention optimization, kernel fusion, and efficient memory usage.

Nice to Have

• Experience supporting inference systems in production.
• Familiarity with robotics or embodied AI workloads.
• Experience building tools for experiment management and researcher productivity.