GPU Infra Engineer for Scalable AI Platform

Kindredventures; San Francisco (CA); On-site; USD 180,000 - 250,000; Full time

Benefits offered by this job

Relocation to San Francisco

Health, dental, and vision insurance (

Regular team events

Learning and growth opportunities

Job summary

fal is building a scalable generative AI infrastructure. We seek an hands-on engineer to manage thousands of servers, automate provisioning, health checks, and recovery, and drive reliability across GPU fleets.

You will develop Python tooling, implement security baselines, and optimize storage and Linux configurations for AI workloads. Relocation to San Francisco is offered and benefits include health/dental/vision.

Qualifications

• 3+ years managing bare-metal and cloud-based server fleets at scale (100+ nodes).
• Strong software engineering skills in Python; production tooling, not scripts.
• Deep Linux systems knowledge: boot process, kernel tuning, networking, storage, systemd.

Responsibilities

• Build and maintain Python fleet tracking system for servers: provisioning, health, RMAs, pricing, availability.
• Develop tooling to automate provisioning, health checks, GPU diagnostics, recovery, and alerting.
• Create metrics, dashboards, and alerting for hardware health across the fleet (GPU errors, disk, network).
• Leverage AI to automate alerting and recovery processes.
• Implement OS-level security: hardening baselines, SELinux/AppArmor policies, SSH key management, vulnerability scanning.
• Manage and optimize storage systems for model weights, checkpoints, scratch space (NVMe, NFS, Lustre/GPFS).
• Tune Linux for AI workloads: kernel, NUMA, CPU pinning, hugepages, I/O schedulers, NVIDIA driver stack.
• Develop a suite of automated error detection and recovery processes.
• Work with partners to solve technical issues.

Skills

Python

Linux systems

Infrastructure as code

System monitoring

Security hardening

Tools

Ansible

Terraform

cloud-init

Job description

fal is building a scalable generative AI infrastructure. We seek an hands-on engineer to manage thousands of servers, automate provisioning, health checks, and recovery, and drive reliability across GPU fleets.

You will develop Python tooling, implement security baselines, and optimize storage and Linux configurations for AI workloads. Relocation to San Francisco is offered and benefits include health/dental/vision.