R Open role · via a16z Speedrun 1d ago Senior Staff Engineer - Infrastructure Systems Runware (Picfinder) US INFRA Express interest →

JoinStartup

California

Remote

USD 180,000 - 260,000

Full time

3 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Runware (Picfinder) seeks a Senior Staff Engineer to own the GPU software stack across firmware, kernel, NVIDIA driver, CUDA runtime and container toolkit. You will ensure stable, scalable GPU deployments in California with remote-friendly workflows.

You will qualify drivers, chase CUDA releases, diagnose GPU failures, and build automated remediation and telemetry. You will collaborate with ops, SRE, and inference teams to keep GPUs healthy and serving at scale.

Qualifications

  • Extensive experience with GPU software stack components.
  • Deep Linux kernel and driver knowledge for production systems.
  • Proficiency in Python and at least one systems language (C/C++/Rust/Go).
  • Experience running GPUs under Kubernetes and bare metal at scale.

Responsibilities

  • Own GPU software baseline across firmware, driver, CUDA, and container tools.
  • Qualify drivers/CUDA releases across generations and workloads.
  • Diagnose and root-cause GPU failures; build robust diagnostics and tests.
  • Make GPU health visible with telemetry and automated remediation.
  • Expose GPUs to workloads via container runtimes, plugins, and MIG policies.
  • Collaborate with ops and SRE teams for Bring-up of new GPU generations.

Skills

NVIDIA stack
CUDA runtime
Linux kernel
Device drivers
Python
C/C++
Rust/Go
Kubernetes
DCGM
NVML
Container Toolkit

Tools

Docker
Kubernetes
PCIe
NVLink
NVSwitch

Job description

Runware (Picfinder)# Senior Staff Engineer - Infrastructure Systems Remote-friendly California, United StatesAT A GLANCESenior Staff Engineer owning the GPU software stack (firmware, kernel, CUDA, containers, Kubernetes) across production infrastructure at scaleLEVELstaff · 12+ yrsKEY SKILLSNVIDIA GPU software stackCUDA runtime and librariesLinux kernel internalsDevice driversPythonC/C++RustGoKubernetesContainer runtimePCIeNVLinkDCGMPyTorchWHAT YOU'LL DO* Own versioned GPU software baseline combining kernel, NVIDIA driver, firmware, CUDA runtime and container toolkit* Qualify every driver and CUDA release against GPU generations, frameworks and model families* Debug GPU failures to root cause: Xid errors, CUDA errors, memory issues, hardware faults* Build GPU acceptance tests, diagnostics and burn-in suites for node qualification* Make GPU health visible and self-healing through telemetry collection and automated remediation* Own GPU exposure to workloads: container runtime, device plugins, GPU Operator, MIG policies, tenant isolation* Bring up new GPU generations and pods with hardware and operations teams* Characterize hardware performance with real inference workloads across GPU generationsTHE ROLEAs Senior Staff Engineer, Infrastructure Systems, you will own the GPU software stack on every Runware node: the firmware, the kernel and NVIDIA driver, the CUDA runtime and libraries, the container runtime, and the way Kubernetes and our inference services see and share a GPU. When a GPU misbehaves anywhere in the fleet, you will find out why and write the software that keeps it from happening again. This is a senior staff-level individual contributor role for someone who knows GPUs from the driver up and still writes code every day. One month that means qualifying a new driver and CUDA release against every model family we serve. The next it is the GPU part of the acceptance suite that every node must pass, a detector that catches a failing GPU from its Xid and ECC history before a customer does, the bring-up of a new GPU generation, or a CUDA error that turns out to start in firmware, a PCIe link or a kernel update. You will work directly with the node operations, Managed Kubernetes, inference performance and Sonic Inference Engine teams, and be the person each of them calls when the problem sits below the framework. Success means that every GPU in every pod runs a known, tested combination of firmware, driver and CUDA, that upgrades roll across the fleet without failed inferences, that GPU faults are detected, classified and remediated automatically, and that a new GPU generation goes from delivery to serving traffic in days. What you'll do* Own the GPU software baseline. Versioned, reproducible combinations of kernel, NVIDIA driver, firmware, CUDA runtime and libraries (cuDNN, cuBLAS, NCCL, TensorRT) and container toolkit, tested before they roll and rolled across the fleet without downtime.* Qualify every driver and CUDA release. Keep the compatibility matrix across GPU generations, frameworks and model families, and catch regressions in correctness, performance and memory use before they reach production.* Take GPU failures to root cause: Xid errors, CUDA \"unknown error\", cuBLAS handle failures, out-of-memory and fragmentation, GPUs that fall off the bus. Separate hardware, firmware, driver, runtime and application causes, and prove which one it is.* Build GPU acceptance and diagnostics: the burn-in, DCGM diagnostics, memory and interconnect tests and real-workload tests that gate every node and rack before it serves traffic, and the reports customers receive at handover.* Make GPU health visible and self-healing. Pull telemetry from DCGM, the driver, the kernel log and the BMC into one trusted view, classify faults, and automate the drain, reset and replacement paths.* Own how GPUs are exposed to workloads: the container runtime, device plugins and the GPU Operator, MIG and sharing policies, and isolation between tenants, on bare metal and on Kubernetes.* Bring up new GPU generations and new pods with the hardware and node operations teams: first driver and firmware, NVLink and NVSwitch validation, and the automation that makes the next bring-up take days.* Characterize hardware with real workloads. Benchmark image, video, audio and LLM inference on each GPU generation and software stack, and feed the results into placement, pricing and the next purchase.* Work with NVIDIA and upstream projects. File driver and firmware bugs with reproducible cases and see them through, track what each release changes, and contribute fixes to the open-source tools we depend on.* Use AI to accelerate the work: agents for log and Xid triage, test generation, release qualification and runbook automation, so that the team's output grows faster than its headcount.Requirements What we're looking for* 12+ years building and operating systems software, with several of those on GPU software running in production at scale.* Deep, hands-on knowledge of the NVIDIA software stack: the kernel driver and its modules, the CUDA runtime and libraries, NVML, DCGM and the Container Toolkit, and how their versions interact.* A record of taking GPU failures to root cause across hardware, firmware, driver, runtime and application: Xid errors, ECC and row remapping, PCIe and NVLink errors, thermal and memory faults.* Strong Linux internals: the kernel, device drivers and modules, PCIe, NUMA and IOMMU, cgroups and containers, and the tools (perf, eBPF, ftrace) that show what a machine is really doing.* You write code every day. Fluent in Python and in at least one systems language (C, C++, Rust or Go), and comfortable enough in CUDA to write a test or reproduce a bug yourself.* Experience running GPUs under Kubernetes and on bare metal: device plugins, the GPU Operator, MIG, image builds and provisioning at scale.* Working knowledge of how inference frameworks use a GPU (PyTorch, TensorRT, ONNX Runtime): memory allocation, compilation and caches, and multi-GPU execution.* Clear technical writing. Your design documents, postmortems and ticket updates can be acted on by someone who was not in the room.* You use AI to accelerate your own output every day, and you have a clear view of how a small team armed with these tools should operate differently.* Comfortable owning outcomes in a fast-moving, ambiguous environment, picking the problem that matters most and being accountable for the result.Nice to have* Time spent building GPU drivers, CUDA libraries or GPU system software at a GPU vendor, a cloud provider or an HPC centre.* Experience with Blackwell-generation hardware (B200, GB200/GB300 NVL72), NVLink and NVSwitch topologies, and liquid-cooled racks.* Multi-GPU and multi-node execution: NCCL, GPUDirect, and RDMA over RoCE or InfiniBand.* Firmware qualification across BIOS, BMC, NIC and GPU, and acceptance testing with OEMs and ODMs.* Inference serving internals: PyTorch, TensorRT-LLM, vLLM, SGLang or custom diffusion and video pipelines, and how they load, schedule and fail.* Experience with non-NVIDIA accelerators (AMD or custom silicon) and their software stacksSourced from the a16z Speedrun talent network — apply on Speedrun.THIS ROLE IS FORSystems ArchitectNot sure if that's you? Find your archetype free →APPLY FOR THIS ROLEApply on Runware (Picfinder) →
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Staff Engineer - Infrastructure Systems
Senior Staff Engineer - Infrastructure Systems

Runware • United States

On-site
USD 250,000 - 320,000
Senior Staff GPU Infra Engineer - AI Supercomputer Scale
Senior Staff GPU Infra Engineer - AI Supercomputer Scale

Runware • United States

Remote
USD 250,000 - 400,000
Stock options
Remote-first culture
Flexible hours
+1
Senior Staff Engineer - Infrastructure Systems
Senior Staff Engineer - Infrastructure Systems

present of AI • California

On-site
USD 180,000 - 240,000
Senior Staff Engineer - Infrastructure Systems
Senior Staff Engineer - Infrastructure Systems

Runware • California

Remote
USD 180,000 - 300,000
Generous paid time off
Meaningful stock options
Flexible hours
+2
Senior Staff Engineer - Infrastructure Systems
Senior Staff Engineer - Infrastructure Systems

Speedrun Talent Network • California

Remote
USD 180,000 - 240,000
Generous PTO
Stock options
Flexible hours
+2
Member of Technical Staff, GPU & Distributed Systems
Member of Technical Staff, GPU & Distributed Systems

Reactor • San Francisco (CA)

On-site
USD 160,000 - 230,000
Member of Technical Staff (Software Engineer, GPU Cluster Infrastructure)
Member of Technical Staff (Software Engineer, GPU Cluster Infrastructure)

Perplexity • United States

On-site
USD 180,000 - 240,000
Member of Technical Staff (Software Engineer, GPU Cluster Infrastructure)
Member of Technical Staff (Software Engineer, GPU Cluster Infrastructure)

Perplexity • United States

Remote
USD 180,000 - 320,000
Senior Manager, Software Engineering - Rack Scale Systems Infrastructure
Senior Manager, Software Engineering - Rack Scale Systems Infrastructure

NVIDIA Corporation • Seattle (WA)

On-site
USD 248,000 - 391,000
Principal Systems Engineer, RELIC Compute Architecture
Principal Systems Engineer, RELIC Compute Architecture

Rune • Mountain View (CA)

Hybrid
USD 320,000 - 520,000
Hybrid work schedule