ML Training Performance Engineer: Scalable Foundation Models

Zoox; Foster City (CA); On-site; USD 177,000 - 270,000; Full time

Benefits offered by this job

Paid time off

Zoox Stock Appreciation Rights

Amazon RSUs

Health insurance

Long-term care insurance

Disability insurance

Life insurance

Job summary

Zoox is building autonomous robotaxis and scaling ML training for VLMs, VLA, and foundation models. The ML Platform team enables innovations across distributed training, mixed-precision, and RL-driven improvements.

You will collaborate with autonomy teams to push training efficiency and throughput while integrating with Zoox’s hardware and software stack. The role emphasizes scaling training workloads, profiling bottlenecks, and developing custom kernels.

Qualifications

• 4+ years total experience, 2+ years on large-scale model training platforms.
• Experience with distributed training frameworks (Megatron-LM, TorchTitan).
• Experience with RL frameworks such as Verl/MILES and RL algorithms (GRPO, PPO).
• Experience with collective communication libraries (NCCL) and optimizing communication/compute overlap in multi-node training.
• Experience using profiling tools (NVIDIA Nsight, PyTorch Profiler, or XLA/JAX profiling).
• Proficient in Python or C++.
• Experience with ML compilers, performance tooling, or hardware/software co-design.

Responsibilities

• Design, implement, and operate cutting-edge ML training performance optimization techniques to scale training efficiency.
• Work on RL framework and RL performance optimization for driving foundation model
• Profile and debug training throughput bottlenecks across the compute, compiler, framework, and cluster-networking stack.
• Collaborate closely with ML researchers, data engineers, and hardware engineers to define requirements and align on architectural decisions for large-scale training infrastructure.
• Implement custom kernels and build benchmarks to track and validate training-performance optimizations across the stack.

Skills

Distributed training

Megatron-LM

TorchTitan

NCCL

Python

C++

NVIDIA Nsight

PyTorch Profiler

XLA/JAX

Tools

Verl/MILES

Job description

Zoox is building autonomous robotaxis and scaling ML training for VLMs, VLA, and foundation models. The ML Platform team enables innovations across distributed training, mixed-precision, and RL-driven improvements.

You will collaborate with autonomy teams to push training efficiency and throughput while integrating with Zoox’s hardware and software stack. The role emphasizes scaling training workloads, profiling bottlenecks, and developing custom kernels.