ML Infra Engineer: Scale Large-Scale Training Systems

physicalintelligence; San Francisco (CA); On-site; USD 180,000 - 240,000; Full time

Job summary

Physical Intelligence is seeking an hands-on ML Infrastructure Engineer to scale and optimize our training systems and core model code. You will own critical infrastructure for large-scale training, managing GPU/TPU compute, job orchestration, and building efficient JAX training pipelines.

You’ll collaborate with researchers and model engineers to translate ideas into experiments and production runs, contributing to core training code for new architectures and multimodal models.

Qualifications

• Hands-on experience building ML training infrastructure.
• Proficiency with JAX and PyTorch.
• Familiarity with distributed training and multi-host setups.
• Experience with cloud platforms (GCP, AWS) and orchestration tools.

Responsibilities

• Own training/inference infrastructure including scheduling, checkpointing, and metrics.
• Scale distributed training across TPU and GPU clusters.
• Profile and optimize memory, throughput, and synchronization.
• Build abstractions for launching, monitoring, and reproducing experiments.
• Collaborate with researchers to translate ideas into production training runs.

Skills

Software engineering

Large-scale training

JAX

PyTorch

Distributed training

Cloud platforms

Tools

JAX

PyTorch

Kubernetes

SLURM

GCP TPU/GKE

AWS

Job description

Physical Intelligence is seeking an hands-on ML Infrastructure Engineer to scale and optimize our training systems and core model code. You will own critical infrastructure for large-scale training, managing GPU/TPU compute, job orchestration, and building efficient JAX training pipelines.

You’ll collaborate with researchers and model engineers to translate ideas into experiments and production runs, contributing to core training code for new architectures and multimodal models.