Senior GPU Infrastructure Engineer for AI Research

Far Ai; United States; Remote; USD 180,000 - 240,000; Full time

Job summary

FAR.AI is a nonprofit AI safety research institute. Foundations, its infrastructure and engineering team, builds the compute platform, tools, and automation enabling researchers.

The role spans scheduling, storage, monitoring, and security for large GPU clusters, with emphasis on pre-training and post-training infrastructure and multi-tenancy to keep experiments fast and safe. Work involves collaborating with researchers and engineers, scaling infrastructure as the organization grows, and

Qualifications

• Experience operating large-scale GPU clusters and pre-/post-training workflows.
• Strong background in distributed storage and networking for HPC.
• Proficiency with infrastructure as code and automation tools.

Responsibilities

• Operate the Kubernetes GPU fleet day to day, including upgrades and rollout.
• Own batch scheduling, multi-tenancy, queues, quotas, and priorities.
• Design and manage storage for datasets, checkpoints, and backups.
• Maintain fault-tolerant multi-node training and cluster health.
• Harden platform security, identity, network policy, and sandboxing.
• Onboard new capacity, test providers, and integrate into IaC pipelines.
• Collaborate directly with researchers and engineers across teams.

Skills

Kubernetes

GPU clusters

Batch scheduling

Security posture

Infrastructure as code

Education

BS in Computer Science or related field

Tools

NVIDIA NCCL

Terraform

Kubernetes tooling

Job description

FAR.AI is a nonprofit AI safety research institute. Foundations, its infrastructure and engineering team, builds the compute platform, tools, and automation enabling researchers.

The role spans scheduling, storage, monitoring, and security for large GPU clusters, with emphasis on pre-training and post-training infrastructure and multi-tenancy to keep experiments fast and safe. Work involves collaborating with researchers and engineers, scaling infrastructure as the organization grows, and