Lead Large-Scale GPU Cluster Engineer for AI Research

Linuxcareers; San Francisco (CA); On-site; USD 120,000 - 180,000; Full time

Job summary

Linuxcareers in San Francisco is building AI research infrastructure. You will design, deploy, and operate large-scale GPU clusters powering training, evaluation, and serving for the research team.

The role emphasizes extending orchestration with Kubernetes/Slurm, building unified interfaces, and ensuring reliability with observability. You'll work with researchers to optimize performance and placement.

Qualifications

• Experience operating large-scale GPU clusters.
• Proficient with Kubernetes, Slurm and Docker.
• Strong Linux, networking, storage, and IaC skills.
• Familiar with cloud ML services (GCP/AWS/Azure).
• CUDA/NCCL experience and profiling for distributed workloads.
• Ability to deliver end-to-end from requirements through autonomous execution.

Responsibilities

• Design, deploy, and operate large distributed GPU clusters end-to-end, including provisioning, imaging, upgrades, and capacity planning
• Extend scheduling and orchestration systems such as Kubernetes and Slurm for topology-aware placement, preemption, quotas, and multi-tenancy across training and inference workloads
• Build software that abstracts cluster management and presents a unified, self-serve interface to researchers and engineers
• Own cluster storage and artifact paths for checkpoints and logs, with clear retention and lineage
• Monitor and continuously improve reliability and error recovery; build observability to catch failures before researchers do
• Partner with researchers to unblock large-scale runs and advise on performance and placement trade-offs

Skills

GPU clusters

Kubernetes

Slurm

Docker

Linux systems

Networking

Storage

Infrastructure as code

Cloud platforms

CUDA/NCCL

Observability

End-to-end ownership

Tools

Kubernetes

Slurm

Docker

Job description

Linuxcareers in San Francisco is building AI research infrastructure. You will design, deploy, and operate large-scale GPU clusters powering training, evaluation, and serving for the research team.

The role emphasizes extending orchestration with Kubernetes/Slurm, building unified interfaces, and ensuring reliability with observability. You'll work with researchers to optimize performance and placement.