Staff Engineer, Distributed GPU Clusters & Infra

Causal Labs; San Francisco (CA); On-site; USD 180,000 - 240,000; Full time

Job summary

Causal Labs is building a Large Physics foundation Model and GPU-driven compute environment to enable rapid research iteration at scale. You will design, deploy, and operate massive GPU clusters, extending Kubernetes and Slurm for efficient, multi-tenant workloads.

You will own end-to-end stack from provisioning to observability, collaborating with researchers to optimize performance and placement. A strong systems background and experience with CUDA/NCCL are essential.

Qualifications

• Experience operating large-scale GPU clusters and container orchestration frameworks (e.g. Kubernetes, Slurm, Docker).
• Strong systems background: Linux, networking, storage, infrastructure-as-code.
• Knowledge of cloud platforms (GCP, AWS, or Azure) and their ML/AI service offerings.
• Understanding of monitoring, logging, observability, and version control best practices for ML systems.
• Familiarity with CUDA/NCCL and performance profiling for distributed workloads.
• Owns deliverables end‑to‑end, from requirements through autonomous execution.

Responsibilities

• Design, deploy, and operate large distributed GPU clusters end to end: provisioning, imaging, upgrades, and capacity planning.
• Extend scheduling and orchestration systems for topology-aware placement, preemption, quotas, and multi-tenancy across training and inference workloads.
• Build software that abstracts cluster management and presents a unified, self-serve interface to researchers and engineers.
• Own cluster storage and artifact paths for checkpoints and logs, with clear retention and lineage.
• Monitor and continuously improve reliability and error recovery; build the observability to catch failures before researchers do.
• Partner with researchers to unblock large-scale runs and advise on performance and placement trade-offs.

Skills

Large-scale GPU clusters

Container orchestration

System administration

Performance profiling

Observability

End-to-end ownership

Tools

Kubernetes

Slurm

Docker

Linux

Networking

Cloud platforms (GCP/AWS/Azure)

Job description

Causal Labs is building a Large Physics foundation Model and GPU-driven compute environment to enable rapid research iteration at scale. You will design, deploy, and operate massive GPU clusters, extending Kubernetes and Slurm for efficient, multi-tenant workloads.

You will own end-to-end stack from provisioning to observability, collaborating with researchers to optimize performance and placement. A strong systems background and experience with CUDA/NCCL are essential.