Senior ML Training Performance Architect

Zoox; Foster City (CA); On-site; USD 180,000 - 240,000; Full time

Job summary

Zoox is seeking an experienced ML Platform engineer to own and optimize training performance for large-scale models. The role involves setting technical direction for distributed training, mixed-precision strategies, and RL on SOTA accelerators, with collaboration across Autonomy teams and internal customers.

The role requires leading projects end-to-end, driving multi-quarter roadmaps, and mentoring engineers to raise the team’s technical bar as we expand into new ML domains.

Qualifications

• 7+ years of total experience, incl. 4+ years on large-scale model training platforms.
• Experience with distributed training frameworks (Megatron-LM, TorchTitan) on GPU clusters.
• Proficiency in Python or C++ and strong profiling skills.

Responsibilities

• Lead the design, implementation, and operation of training performance optimizations to scale VLM, VLA, and Foundation model training.
• Own the technical strategy and multi-quarter roadmap for RL framework and RL performance optimization for driving foundation models.
• Drive profiling and root-cause analysis of throughput bottlenecks across compute, compiler, framework, and cluster-networking stack.
• Partner with ML researchers, data engineers, and hardware engineers to define requirements and make architectural decisions for large-scale training infrastructure.
• Build custom kernels and benchmarks that track and validate performance gains across the stack.
• Mentor engineers and raise the team’s technical bar through design reviews and best practices.

Skills

Python

C++

Distributed training

Profiling tools

Leadership

Cross-team communication

Tools

Megatron-LM

TorchTitan

NCCL

Job description

Zoox is seeking an experienced ML Platform engineer to own and optimize training performance for large-scale models. The role involves setting technical direction for distributed training, mixed-precision strategies, and RL on SOTA accelerators, with collaboration across Autonomy teams and internal customers.

The role requires leading projects end-to-end, driving multi-quarter roadmaps, and mentoring engineers to raise the team’s technical bar as we expand into new ML domains.