Senior/Staff Software Engineer, Kubernetes Infrastructure

Fal; United States; Remote; USD 180,000 - 250,000; Full time

Benefits offered by this job

Visa sponsorship

Relocation to San Francisco

Health insurance

Job summary

fal is building the high-performance compute environments for customers, spanning bare-metal servers, GPU-passthrough VMs, Kubernetes and Slurm clusters, and distributed storage. You will shape Linux images, provisioning, and cluster networking to deliver reliable, repeatable environments.

You will drive automation across provisioning, upgrades, and recovery, leveraging AI to accelerate every facet of infrastructure delivery and operations while collaborating with customers and internal teams.

Qualifications

• Experience designing and operating production Linux infrastructure.
• Proficient with Kubernetes on bare metal and container runtimes.
• Familiarity with NVIDIA GPU drivers, GPU telemetry, and GPU acceleration tooling.
• Strong networking fundamentals and distributed storage experience.

Responsibilities

• Design, automate, validate, and deliver the complete lifecycle of customer compute environments—from provisioning through upgrades, recovery, and decommissioning.
• Use AI to automate and accelerate infrastructure delivery and operations.
• Provision dedicated Kubernetes and Slurm clusters for customer workloads.
• Build and maintain Linux images and automated OS provisioning workflows.
• Operate the NVIDIA GPU stack and related tooling.
• Design Kubernetes and data-center networking and configure distributed storage for high-performance workloads.
• Build monitoring, alerting, diagnostics, and automated recovery for customer environments.
• Develop reusable tooling, standards, documentation, and runbooks.
• Collaborate with customers and internal teams to translate workload requirements into infrastructure designs.

Skills

Strong communication

Ownership

Problem solving

Team collaboration

Tools

Kubernetes

Slurm

Linux

NVIDIA GPU stack

Cilium/Calico networking

Ceph/Lustre storage

GPU Operator

Docker/Containerd

Job description

fal is the generative media ecosystem powering the next generation of AI products. We build the infrastructure, tools, and model access that teams need to move from idea to production, and do it at scale without compromise. For developers and enterprises, fal is the foundation that makes generative media not just possible, but practical: a unified platform where high-performance inference, orchestration, and observability come together to unlock new categories of AI-native products.

As generative media reshapes industries across a market projected to grow by hundreds of billions over the next decade, fal is becoming the ecosystem that ambitious teams build on.

About this role:

You will build the high-performance compute environments we deliver to customers. These environments span bare-metal servers, virtual machines with GPU passthrough, Kubernetes and Slurm clusters, distributed storage, and high-speed networking.

You will work across the infrastructure stack — from Linux images and hardware provisioning to cluster networking, GPU performance, observability, and lifecycle automation. The goal is to make every customer environment performant, reliable, isolated, and repeatable.

What you'll do:

• Design, automate, validate, and deliver the complete lifecycle of customer compute environments — from provisioning through upgrades, recovery, and decommissioning

• Use AI aggressively to automate and accelerate every aspect of infrastructure delivery and operations

• Provision dedicated Kubernetes and Slurm clusters tailored to customer workloads

• Build and maintain Linux images and automated OS-provisioning workflows

• Operate the NVIDIA GPU stack: drivers, GPU Operator, NVIDIA Container Toolkit, device plugins, MIG, and GPU monitoring

• Design Kubernetes and data-center networking using Cilium/Calico, MetalLB, VLAN, VXLAN, BGP, and ECMP

• Configure distributed and shared storage for high-performance workloads

• Build monitoring, alerting, diagnostics, and automated recovery for customer environments

• Develop reusable tooling, standards, documentation, and runbooks

• Collaborate with customers and internal teams to translate workload requirements into sound infrastructure designs

Qualifications/Nice to Have:

• 5+ years of experience building and operating production Linux infrastructure

• Strong production experience with Kubernetes on bare metal (bootstrapping, upgrades, HA control planes, etcd, containerd, CNI, CSI, ingress, load-balancing, observability, security, troubleshooting)

• Experience with Linux virtualization: KVM/QEMU, libvirt, VFIO device passthrough

• Experience operating NVIDIA GPUs on Linux and Kubernetes (drivers, container runtimes, device plugins, GPU Operator, GPU telemetry)

• Strong networking fundamentals: TCP/IP, L2/L3, VLANs, routing, packet-level troubleshooting (tcpdump, Wireshark)

• Practical scripting experience

• Experience with configuration-management tools such as Ansible

• Ability to diagnose complex, cross-layer infrastructure issues

• Strong communication and ability to drive technical decisions across teams

• Track record of moving quickly, taking ownership, and continuously improving systems

• Production Slurm experience

• High-performance networking: NVLink/NVSwitch, InfiniBand, RoCEv2, GPUDirect RDMA, NCCL, IMEX

• Hugepages, NUMA, CPU pinning

• SR-IOV, DPDK

• Distributed storage: Ceph, Lustre, Weka

• KubeVirt, OpenStack

• IPsec, WireGuard, Tailscale

• VXLAN, BGP, ECMP

• Bare-metal management: BMC, IPMI, Redfish, PXE/iPXE, Kickstart, cloud-init

• Network automation: NetBox, Nautobot, Nornir

• AI training, inference, or distributed GPU workload infrastructure

• Python or Go proficiency

What we offer at fal

• Interesting and challenging work

• A lot of learning and growth opportunities

• We offer visa sponsorship and will help you relocate to San Francisco.

• Health, dental, and vision insurance (US)

• Regular team events and offsites

Compensation

• $180,000 - $250,000 + equity + comprehensive benefits package

U.S. EQUAL EMPLOYMENT OPPORTUNITY INFORMATION:

fal provides equal employment opportunities to applicants and employees without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, protected veteran status, disability, or any other classification protected by applicable law.