ML Infrastructure Engineer: Scale & Performance

Physical Intelligence; San Francisco (CA); On-site; USD 150,000 - 230,000; Full time

Job summary

Physical Intelligence in San Francisco is building the core ML infrastructure to scale training from prototype to production-grade runs. The ML Infrastructure team owns training/inference systems, scheduling, checkpointing, and metrics collection.

You will scale JAX-based training across TPU and GPU clusters, profile memory usage, improve throughput, and create abstractions for launching and monitoring experiments in close collaboration with researchers.

Responsibilities

• Own training/inference infrastructure: Design, implement, and maintain systems for large-scale model training, including scheduling, job management, checkpointing, and metrics/logging
• Scale distributed training: Work with researchers to scale JAX-based training across TPU and GPU clusters with minimal friction
• Optimize performance: Profile and improve memory usage, device utilization, throughput, and distributed synchronization
• Enable rapid iteration: Build abstractions for launching, monitoring, debugging, and reproducing experiments
• Partner with researchers: Translate research needs into infra capabilities and guide best practices for training at scale
• Contribute to core training code: Evolve JAX model and training code to support new architectures, modalities, and evaluation metrics

Job description

Physical Intelligence in San Francisco is building the core ML infrastructure to scale training from prototype to production-grade runs. The ML Infrastructure team owns training/inference systems, scheduling, checkpointing, and metrics collection.

You will scale JAX-based training across TPU and GPU clusters, profile memory usage, improve throughput, and create abstractions for launching and monitoring experiments in close collaboration with researchers.