Physical Intelligence is developing general-purpose AI for robotics and physical systems. This role owns end-to-end intelligent job scheduling and multi-cluster compute orchestration, designing systems that automatically place large-scale training jobs across heterogeneous GPU/TPU clusters while optimizing utilization, cost, and researcher velocity.
What You'll Do
- Design and build multi-tenant scheduling systems that automatically place training jobs on the best cluster based on hardware requirements, topology, availability, cost, and priority
- Build the control plane managing job lifecycle across diverse clusters with mixed GPU/TPU, multi-generation hardware, on-premises and cloud infrastructure, enabling seamless job migration and failover
- Monitor and optimize GPU/TPU utilization across the entire fleet, implementing priority, preemption, queueing, and fairness policies
- Implement fault detection, automatic recovery, and resilience for long-running multi-node training jobs across thousands of accelerators
- Build observability dashboards, alerting, SLOs, and debugging tools for researchers to understand job status and ensure high scheduling quality
What You Need
- Strong software engineering fundamentals
- Experience building or operating job scheduling and resource management systems at scale
- Experience with large-scale compute clusters, specifically GPU and/or TPU infrastructure
- Familiarity with schedulers and orchestration systems such as SLURM, Kubernetes, GKE, or K3s
- Understanding of how ML training workloads behave including long-running, multi-node, and topology-dependent characteristics
Nice to Have
- Experience building multi-cluster or federated scheduling systems
- Experience with TPU infrastructure including GCP TPU slices, Multislice, and GKE
- Background in cluster resource managers such as Borg, YARN, Mesos, or custom schedulers
- Linux systems engineering, networking, and infrastructure-as-code experience
- Knowledge of NCCL, collective communication, and topology-aware placement
- Experience with capacity planning and cloud cost optimization at scale
- Familiarity with JAX, PyTorch, or similar ML frameworks at the runtime and systems level