Research Engineer: Scalable ML Training Systems

Mind Robotics Inc.; Palo Alto, Northern (CA, KY); Hybrid; USD 180,000 - 300,000; Full time

Job summary

Mind Robotics is seeking a Research Engineer to build core systems for scalable ML model training on real-world industrial data. You will design distributed training pipelines across hundreds of GPUs, enabling fast experimentation and cost-efficient iterations.

Collaborate with modeling teams to accelerate iteration speed, reduce training costs, and develop internal tooling for experiment tracking, monitoring, and deployment.

Qualifications

• Extensive experience building infrastructure for large-scale ML training.
• Experience scaling distributed training across hundreds of GPUs.
• Strong Python, PyTorch/JAX, and optimizer knowledge.

Responsibilities

• Design and implement scalable systems for training large ML models.
• Enable efficient workflows for data ingestion, training, and iteration.
• Develop and optimize distributed training systems across hundreds of GPUs.
• Implement strategies for parallelization, sharding, and efficient compute utilization.
• Improve training efficiency through attention optimization, kernel fusion, and memory management.
• Partner closely with modeling teams to accelerate iteration speed and reduce training costs.
• Build internal tools for experiment tracking, monitoring, and debugging.
• Implement systems for tracking training performance, failures, and resource utilization.
• Debug and resolve bottlenecks across the training stack.
• Provide lightweight infrastructure support for deploying and running models in production environments.
• Optimize inference performance and reliability where needed.
• Support core cloud infrastructure needs for training workloads (without heavy DevOps overhead).
• Manage compute resources efficiently across training jobs.

Skills

Python

PyTorch

JAX

Distributed training

GPU scaling

Memory optimization

Research collaboration

Tools

Linux

Bash scripting

Experiment tracking

Job description

Mind Robotics is seeking a Research Engineer to build core systems for scalable ML model training on real-world industrial data. You will design distributed training pipelines across hundreds of GPUs, enabling fast experimentation and cost-efficient iterations.

Collaborate with modeling teams to accelerate iteration speed, reduce training costs, and develop internal tooling for experiment tracking, monitoring, and deployment.