ML Infra Engineer, Compute

Linuxconfig

San Francisco (CA)

On-site

USD 180,000 - 240,000

Full time

48 hours ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Physical Intelligence in San Francisco is seeking an experienced engineer to design and operate end-to-end job scheduling and multi-cluster orchestration for large-scale AI training across GPU/TPU clusters.

You will optimize utilization, cost, and researcher velocity by building scalable control planes, fault tolerance, and observability dashboards across on-prem and cloud environments. Strong software fundamentals and experience with SLURM, Kubernetes, or similar schedulers are essential.

Qualifications

  • Strong software engineering fundamentals.
  • Experience building or operating scheduling systems at scale.
  • Experience with large-scale compute clusters, specifically GPU/TPU infrastructure.
  • Familiarity with schedulers and orchestration systems such as SLURM, Kubernetes, GKE, or K3s.
  • Understanding of ML training workloads including long-running, multi-node, and topology-dependent characteristics.

Responsibilities

  • Design and build multi-tenant scheduling systems that automatically place training jobs on the best cluster based on hardware requirements, topology, availability, cost, and priority
  • Build the control plane managing job lifecycle across diverse clusters with mixed GPU/TPU, multi-generation hardware, on-premises and cloud infrastructure, enabling seamless job migration and failover
  • Monitor and optimize GPU/TPU utilization across the entire fleet, implementing priority, preemption, queueing, and fairness policies
  • Implement fault detection, automatic recovery, and resilience for long-running multi-node training jobs across thousands of accelerators
  • Build observability dashboards, alerting, SLOs, and debugging tools for researchers to understand job status and ensure high scheduling quality

Skills

Software engineering fundamentals
Systems at scale
GPU/TPU infrastructure
Schedulers/orchestration
ML training workloads understanding

Tools

SLURM
Kubernetes
GKE
K3s

Job description

Physical Intelligence is developing general-purpose AI for robotics and physical systems. This role owns end-to-end intelligent job scheduling and multi-cluster compute orchestration, designing systems that automatically place large-scale training jobs across heterogeneous GPU/TPU clusters while optimizing utilization, cost, and researcher velocity.

What You'll Do
  • Design and build multi-tenant scheduling systems that automatically place training jobs on the best cluster based on hardware requirements, topology, availability, cost, and priority
  • Build the control plane managing job lifecycle across diverse clusters with mixed GPU/TPU, multi-generation hardware, on-premises and cloud infrastructure, enabling seamless job migration and failover
  • Monitor and optimize GPU/TPU utilization across the entire fleet, implementing priority, preemption, queueing, and fairness policies
  • Implement fault detection, automatic recovery, and resilience for long-running multi-node training jobs across thousands of accelerators
  • Build observability dashboards, alerting, SLOs, and debugging tools for researchers to understand job status and ensure high scheduling quality
What You Need
  • Strong software engineering fundamentals
  • Experience building or operating job scheduling and resource management systems at scale
  • Experience with large-scale compute clusters, specifically GPU and/or TPU infrastructure
  • Familiarity with schedulers and orchestration systems such as SLURM, Kubernetes, GKE, or K3s
  • Understanding of how ML training workloads behave including long-running, multi-node, and topology-dependent characteristics
Nice to Have
  • Experience building multi-cluster or federated scheduling systems
  • Experience with TPU infrastructure including GCP TPU slices, Multislice, and GKE
  • Background in cluster resource managers such as Borg, YARN, Mesos, or custom schedulers
  • Linux systems engineering, networking, and infrastructure-as-code experience
  • Knowledge of NCCL, collective communication, and topology-aware placement
  • Experience with capacity planning and cloud cost optimization at scale
  • Familiarity with JAX, PyTorch, or similar ML frameworks at the runtime and systems level
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

ML Infra Engineer, Compute
ML Infra Engineer, Compute

Physical Intelligence • San Francisco (CA)

On-site
USD 180,000 - 260,000
ML Infra Engineer: Intelligent Scheduling for Large-Scale AI
ML Infra Engineer: Intelligent Scheduling for Large-Scale AI

Physical Intelligence • San Francisco (CA)

On-site
USD 180,000 - 260,000
ML Infra Scheduler & Multi-Cluster Orchestrator
ML Infra Scheduler & Multi-Cluster Orchestrator

Linuxconfig • San Francisco (CA)

On-site
USD 180,000 - 240,000
Member of Technical Staff — Compute Cluster
Member of Technical Staff — Compute Cluster

Kindredventures • San Francisco (CA)

On-site
USD 140,000 - 230,000
Member of Technical Staff
Member of Technical Staff

Harrison Clarke • San Francisco (CA)

On-site
USD 180,000 - 280,000
Member of Technical Staff - ML Infra
Member of Technical Staff - ML Infra

Kindredventures • San Francisco (CA)

On-site
USD 160,000 - 220,000
Software Engineer: ML Infra
Software Engineer: ML Infra

Generalist • Somerville (MA), San Mateo (CA)

On-site
USD 120,000 - 160,000
Infrastructure / Cluster Engineer
Infrastructure / Cluster Engineer

Linuxconfig • San Francisco (CA)

On-site
USD 120,000 - 160,000
ML Infra Engineer (Data Systems)
ML Infra Engineer (Data Systems)

Physical Intelligence • San Francisco (CA)

On-site
USD 180,000 - 240,000
ML Infrastructure Engineer
ML Infrastructure Engineer

Sovyren • United States

Remote
USD 150,000 - 230,000