Senior GPU Cluster Infra Engineer | Remote

AISafety; Berkeley (CA); Hybrid; USD 120,000 - 180,000; Full time

Benefits offered by this job

Health Insurance

401(k) match

PTO 25 days per year

Bereavement leave

WFH stipend

Catered meals (Berkeley Office)

Job summary

FAR.AI seeks an experienced infrastructure engineer to manage and scale its GPU cluster infrastructure, spanning scheduling, storage, and security. You will collaborate with researchers and other engineers to ensure high availability and efficient workload execution on large-scale GPU clusters.

The role emphasizes hands-on work with Kubernetes, batch systems, and secure multi-tenant environments, contributing to frontier AI research while maintaining robust production operations.

Qualifications

• 3+ years in systems or infrastructure engineering on production Linux
• Experience running GPU workloads with a batch layer
• Own infrastructure as code and observability for a production fleet
• Proficient programming in Python, Go, Rust, or C++
• Able to write clearly for engineers, researchers, and providers

Responsibilities

• Operate the Kubernetes GPU fleet day to day, including node lifecycle, upgrades, and capacity planning
• Own batch scheduling, quotas, priorities, preemption, and fair share across teams
• Design and run storage under the fleet with high-performance shared filesystems and object storage
• Maintain fault tolerance for multi-node training, diagnose NCCL/fabric issues, and implement checkpoint/restart patterns
• Harden the platform with identity, network policy, secrets, and sandboxing for AI agents

Skills

Kubernetes admin

GPU workloads

Infrastructure as code

Programming (Python/Go/C++)

System scalability

Tools

Terraform

Ansible

Helm

ArgoCD

Prometheus

Job description

FAR.AI seeks an experienced infrastructure engineer to manage and scale its GPU cluster infrastructure, spanning scheduling, storage, and security. You will collaborate with researchers and other engineers to ensure high availability and efficient workload execution on large-scale GPU clusters.

The role emphasizes hands-on work with Kubernetes, batch systems, and secure multi-tenant environments, contributing to frontier AI research while maintaining robust production operations.