ML Infra Engineer: Intelligent Scheduling for Large-Scale AI

Physical Intelligence

San Francisco (CA)

On-site

USD 180,000 - 260,000

Full time

2 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Physical Intelligence is seeking an experienced specialist to own intelligent job scheduling and multi-cluster orchestration for GPU/TPU fleets. You will design multi-tenant scheduling, placement logic, and the control plane across on-prem/cloud clusters, enabling seamless job migration and high utilization.

You will partner with researchers and ML researchers to optimize resource allocation, fault tolerance, and scale for long-running multi-node training and edge deployment.

Qualifications

  • Strong software eng. fundamentals and system design skills.
  • Experience building or operating scheduling/resource mgmt systems at scale.
  • Experience with large-scale compute clusters (GPU/TPU).
  • Familiarity with schedulers and orchestration systems (SLURM, Kubernetes, GKE, K3S).
  • Comfort with resource allocation, bin-packing, priority scheduling, multi-tenancy.
  • Understanding ML training workloads: long-running, multi-node, stragglers, topology-aware.

Responsibilities

  • Own Intelligent Job Scheduling and Placement across multi-tenant clusters.
  • Scale multi-cluster orchestration; enable failover and re-scheduling.
  • Optimize accelerator utilization with fairness and cost-efficiency policies.
  • Ensure scaling and stability for thousands of accelerators across clusters.
  • Support inference and robot deployment workloads via scheduling.
  • Improve observability with dashboards, alerts, and SLOs for researchers.

Skills

Software engineering fundamentals
Experience building or operating job-s
Large-scale compute clusters (GPU/TPU)
ML training workloads understanding
Ownership of end-to-end systems
Collaborative with researchers

Tools

SLURM
Kubernetes
GKE
K3S
Borg
YARN
Mesos

Job description

Physical Intelligence is seeking an experienced specialist to own intelligent job scheduling and multi-cluster orchestration for GPU/TPU fleets. You will design multi-tenant scheduling, placement logic, and the control plane across on-prem/cloud clusters, enabling seamless job migration and high utilization.

You will partner with researchers and ML researchers to optimize resource allocation, fault tolerance, and scale for long-running multi-node training and edge deployment.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

ML Infra Scheduler & Multi-Cluster Orchestrator
ML Infra Scheduler & Multi-Cluster Orchestrator

Linuxconfig • San Francisco (CA)

On-site
USD 180,000 - 240,000
ML Infra Engineer, Compute
ML Infra Engineer, Compute

Linuxconfig • San Francisco (CA)

On-site
USD 180,000 - 240,000
ML Infra Engineer, Compute
ML Infra Engineer, Compute

Physical Intelligence • San Francisco (CA)

On-site
USD 180,000 - 260,000
ML Infra Engineer: GPU Fleet & Inference Orchestrator
ML Infra Engineer: GPU Fleet & Inference Orchestrator

Generalist • San Francisco (CA)

On-site
USD 120,000 - 160,000
ML Infra Engineer: GPU Fleet & Inference Orchestration
ML Infra Engineer: GPU Fleet & Inference Orchestration

Generalist • Somerville (NJ)

On-site
USD 140,000 - 190,000
ML Infrastructure Engineer: Kubernetes Scheduler
ML Infrastructure Engineer: Kubernetes Scheduler

Isomorphic Labs • London (KY)

Hybrid
USD 180,000 - 240,000
Hybrid work model
ML Infrastructure Engineer: Scale & Performance
ML Infrastructure Engineer: Scale & Performance

Physical Intelligence • San Francisco (CA)

On-site
USD 150,000 - 230,000
ML Infra Engineer - Scale GPU Clusters & Simulation
ML Infra Engineer - Scale GPU Clusters & Simulation

Zeromatter • Mountain View (CA)

On-site
USD 100,000 - 300,000
Equity
Medical, dental, vision insurance
ML Infra Engineer — Scale AI Platform with Kubernetes
ML Infra Engineer — Scale AI Platform with Kubernetes

Physical Intelligence • San Francisco (CA)

On-site
USD 180,000 - 210,000
Senior Infrastructure Engineer - GPU Clusters & AI Pipelines
Senior Infrastructure Engineer - GPU Clusters & AI Pipelines

General Intuition • New York (NY)

On-site
USD 150,000 - 210,000