ML Infra Scheduler & Multi-Cluster Orchestrator

Linuxconfig

San Francisco (CA)

On-site

USD 180,000 - 240,000

Full time

48 hours ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Physical Intelligence in San Francisco is seeking an experienced engineer to design and operate end-to-end job scheduling and multi-cluster orchestration for large-scale AI training across GPU/TPU clusters.

You will optimize utilization, cost, and researcher velocity by building scalable control planes, fault tolerance, and observability dashboards across on-prem and cloud environments. Strong software fundamentals and experience with SLURM, Kubernetes, or similar schedulers are essential.

Qualifications

  • Strong software engineering fundamentals.
  • Experience building or operating scheduling systems at scale.
  • Experience with large-scale compute clusters, specifically GPU/TPU infrastructure.
  • Familiarity with schedulers and orchestration systems such as SLURM, Kubernetes, GKE, or K3s.
  • Understanding of ML training workloads including long-running, multi-node, and topology-dependent characteristics.

Responsibilities

  • Design and build multi-tenant scheduling systems that automatically place training jobs on the best cluster based on hardware requirements, topology, availability, cost, and priority
  • Build the control plane managing job lifecycle across diverse clusters with mixed GPU/TPU, multi-generation hardware, on-premises and cloud infrastructure, enabling seamless job migration and failover
  • Monitor and optimize GPU/TPU utilization across the entire fleet, implementing priority, preemption, queueing, and fairness policies
  • Implement fault detection, automatic recovery, and resilience for long-running multi-node training jobs across thousands of accelerators
  • Build observability dashboards, alerting, SLOs, and debugging tools for researchers to understand job status and ensure high scheduling quality

Skills

Software engineering fundamentals
Systems at scale
GPU/TPU infrastructure
Schedulers/orchestration
ML training workloads understanding

Tools

SLURM
Kubernetes
GKE
K3s

Job description

Physical Intelligence in San Francisco is seeking an experienced engineer to design and operate end-to-end job scheduling and multi-cluster orchestration for large-scale AI training across GPU/TPU clusters.

You will optimize utilization, cost, and researcher velocity by building scalable control planes, fault tolerance, and observability dashboards across on-prem and cloud environments. Strong software fundamentals and experience with SLURM, Kubernetes, or similar schedulers are essential.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

ML Infra Engineer: Intelligent Scheduling for Large-Scale AI
ML Infra Engineer: Intelligent Scheduling for Large-Scale AI

Physical Intelligence • San Francisco (CA)

On-site
USD 180,000 - 260,000
ML Infra Engineer, Compute
ML Infra Engineer, Compute

Linuxconfig • San Francisco (CA)

On-site
USD 180,000 - 240,000
ML Infra Engineer, Compute
ML Infra Engineer, Compute

Physical Intelligence • San Francisco (CA)

On-site
USD 180,000 - 260,000
ML Infra Engineer — Scale AI Platform with Kubernetes
ML Infra Engineer — Scale AI Platform with Kubernetes

Physical Intelligence • San Francisco (CA)

On-site
USD 180,000 - 210,000
ML Infrastructure Engineer: Kubernetes Scheduler
ML Infrastructure Engineer: Kubernetes Scheduler

Isomorphic Labs • London (KY)

Hybrid
USD 180,000 - 240,000
Hybrid work model
Senior GPU Infrastructure Engineer — HPC & Clusters
Senior GPU Infrastructure Engineer — HPC & Clusters

Prime Intellect AI • San Francisco (CA)

On-site
USD 150,000 - 300,000
ML Infrastructure Engineer: Scale & Performance
ML Infrastructure Engineer: Scale & Performance

Physical Intelligence • San Francisco (CA)

On-site
USD 150,000 - 230,000
ML Infra Engineer - Distributed Training & Orchestration
ML Infra Engineer - Distributed Training & Orchestration

MaxIT Consulting - Max Corporate Group • San Francisco (CA)

Hybrid
USD 150,000 - 190,000
ML Infra Engineer - Scale GPU Clusters & Simulation
ML Infra Engineer - Scale GPU Clusters & Simulation

Zeromatter • Mountain View (CA)

On-site
USD 100,000 - 300,000
Equity
Medical, dental, vision insurance
Senior GPU Infrastructure Engineer: Schedulers & ML Infra
Senior GPU Infrastructure Engineer: Schedulers & ML Infra

Career Movement • San Francisco (CA)

On-site
USD 180,000 - 240,000