ML Infra Engineer: GPU Orchestration & Observability

Autolab; San Francisco (CA); On-site; USD 150,000 - 210,000; Full time

Job summary

Autolab is building the orchestration and storage backbone for thousands of ML experiments in a production-grade environment. You will design and operate the scheduling layer, optimize GPU utilization, and own the artifact storage and observability for large-scale workflows.

You’ll work with Python and systems languages (Go, Rust, or C++) and gain hands-on experience with Kubernetes, Slurm, and Ray to support diverse GPU fleets and HPC workloads.

Qualifications

• Experience building distributed systems in production: schedulers, queues, or storage.
• Strong Python plus a systems language (Go, Rust, or C++).
• Hands-on experience with cluster tooling such as Kubernetes, Slurm, or Ray.

Responsibilities

• Design and operate the orchestration layer that schedules thousands of concurrent experiments across heterogeneous GPU fleets.
• Build GPU scheduling that keeps clusters near full utilization, from a single lab 3090 to multi-node H100 clusters.
• Own artifact storage for checkpoints, datasets, and logs: versioned, deduplicated, and fast to fetch.
• Build the observability pipeline that ingests training telemetry at high volume without slowing runs down.
• Make experiment launches reproducible: environments, dependencies, and data pinned by default.
• Set the engineering practices, CI, deployment, on-call, that the team grows into.

Skills

Python

Go

Rust

C++

Tools

Kubernetes

Slurm

Ray

Job description

Autolab is building the orchestration and storage backbone for thousands of ML experiments in a production-grade environment. You will design and operate the scheduling layer, optimize GPU utilization, and own the artifact storage and observability for large-scale workflows.

You’ll work with Python and systems languages (Go, Rust, or C++) and gain hands-on experience with Kubernetes, Slurm, and Ray to support diverse GPU fleets and HPC workloads.