Senior Storage Engineer

Runware; Palo Alto (CA); Remote; USD 170,000 - 250,000; Full time

Benefits offered by this job

Stock options

Remote-first culture

Generous paid time off

Job summary

Runware is building the AI Factory with a distributed storage fabric across the US and Europe. We seek a Senior Storage Engineer to own the storage architecture for models, caches and customer data, and to design the data path from object storage to NVMe and GPU memory.

You will implement the first version and set the storage standard across pods. You will work hands-on with inference, platform and hardware teams, optimize load latency, and automate benchmarks using AI-driven approaches to scale

Qualifications

• 8+ years designing and running storage at scale.
• Deep Linux storage expertise: block layer, NVMe, filesystems.
• Production experience with Lustre, Ceph or GPFS.
• Proficiency in Go, Rust or C/C++ and Python.
• Experience supporting GPU AI workloads is a plus.

Responsibilities

• Own end-to-end storage architecture for the fleet and models.
• Design and build the model distribution path across pods.
• Profile disk to GPU memory path and reduce latency.
• Benchmark, operate and tune storage systems across hardware.
• Set reliability, durability and data residency targets.
• Automate benchmarks, runbooks and tooling with AI.
• Collaborate with inference engine, platform and hardware teams.
• Ensure secure, compliant data handling in multi-region deployments.

Skills

Storage systems design

Linux systems engineering

Go / Rust / C/C++

Python scripting

AI-powered tooling

Travel readiness

Tools

Lustre

GPFS

Ceph

S3-compatible object storage

NVMe / NVMe-oF

Prometheus / Grafana

Job description

Runware is the AI inference platform that builds and runs its own infrastructure, end to end. Our Sonic Inference Engine has served more than 20 billion requests for over 1 million developers and 500 million end users, with 400,000+ models behind a single integration, at up to 10x lower cost per token than traditional providers.

We are now building the AI Factory that runs it. The Sonic Inference Pod packs 1 MW of custom-designed compute into a 20-foot container with closed-loop, water-free cooling, sited wherever power exists. Pods are live in the US and Europe, and we are scaling to dozens of pods and tens of thousands of GPUs, all operated as one distributed fabric that reroutes traffic the moment a pod goes offline (announcement).

Every one of those 400,000 models has to be on the right GPU, in the right pod, in seconds. Storage is what makes that possible. That is the job.

The role

As Senior Storage Engineer, you will own the storage architecture of Runware's AI supercomputer: how model weights, caches and customer data move from object storage on two continents to the NVMe and GPU memory of the node that needs them. You will design the data path, build the first version with your own hands, and set the standard for how storage is run across every pod.

This is a hands-on senior individual-contributor role. You will specify the systems, benchmark them on real hardware, find the reason a 70 GB checkpoint took four minutes to load instead of twenty seconds, and then fix it at the root so it never happens again. You will work directly with our inference engine, platform and hardware teams, with our storage and NVMe vendors, and with the site teams deploying pods.

Success means a cold model loads onto a GPU anywhere in the fleet in seconds, a new pod has its working set warm before it takes traffic, and a disk, node or whole site failing is invisible to the developer calling our API.

What You'll Do

• Own the end-to-end storage architecture for the fleet: the durable object store that holds 400,000+ models, the per-pod high-throughput tier that feeds GPUs, the node-local NVMe cache, and the replication and placement logic that ties them together across the US and Europe
• Design and build the model distribution path. Decide how weights get from the source of truth to a pod, how they are chunked, deduplicated, prefetched and pinned, and how a new pod warms its working set before it takes production traffic
• Drive model load latency down and keep it down. Profile the full path from disk to GPU memory, including the kernel, file system, NVMe-oF or RDMA transport, GPUDirect Storage and the inference engine's loader, and remove the bottleneck wherever it sits
• Select, benchmark and operate the storage systems themselves: parallel and distributed file systems, object storage, NVMe drives and enclosures, and the software that manages them. Validate every choice on our hardware before it ships to a pod
• Set and hit reliability and durability targets. Own capacity planning, monitoring, alerting, drive health and failure prediction, rebuild behaviour, backup and recovery, and on-call for the storage layer
• Make storage a routine part of pod bring-up: a documented, automated playbook so that the storage tier of the next 1 MW pod is provisioned, validated and warm in hours, not weeks
• Partner with inference engine, platform and hardware engineering so that what we design is operable at scale, and feed what you learn from production back into the next generation of servers and pods
• Use AI to accelerate the work: automate benchmarks, runbooks, triage and tooling with LLM-based agents, so that a very small team can run storage for a very large fleet
• Own data residency and the operational side of storage security: encryption at rest, access control, retention and in-country placement for sovereign deployments

Requirements

What we're looking for

• 8+ years in storage or systems engineering, with several of those designing and running storage at a company that operates its own hardware at scale
• A track record of building storage systems, not just operating what someone else built. You have taken a design from the whiteboard to petabytes in production and know what goes wrong between the two
• Deep, current, hands-on Linux storage expertise: the block layer, io_uring and NVMe queues, file systems such as XFS, ext4 and ZFS, the page cache, and the tools to see exactly where an I/O spends its time
• Real experience with at least one parallel or distributed file system in production (Lustre, WEKA, VAST, GPFS, Ceph or similar) and with S3-compatible object storage at petabyte scale
• Strong understanding of NVMe, NVMe-oF, RDMA transports and the networking that storage rides on. You can tell a saturated PCIe link from a slow drive from a misconfigured switch
• Fluent in a systems language (Go, Rust or C/C++) and Python, and comfortable writing the tooling, benchmarks and automation yourself rather than waiting for someone else to
• You use AI to accelerate your own output every day, for scripting, benchmarking, analysis, documentation and debugging, and you have a clear view of how a small team armed with these tools should operate differently
• Comfortable owning outcomes in a fast-moving, ambiguous environment, making decisions with incomplete information and being accountable for latency, durability and cost
• Able to travel to sites in the US and Europe as needed

Nice to have

• Experience feeding large GPU fleets for AI training or inference: GPUDirect Storage, NCCL-aware data loading, model weight caching and the loaders in vLLM, SGLang, TensorRT-LLM or similar engines
• Hands-on experience with InfiniBand or RoCE fabrics and 400G+ Ethernet from the storage side
• Experience with content-addressed or deduplicated storage, erasure coding, and multi-region replication and consistency trade-offs
• Kernel or driver contributions, or deep familiarity with SPDK, DPDK or user-space storage stacks
• Experience with Kubernetes CSI, Ceph or Rook on bare metal, and an automation stack such as Ansible and Terraform with Prometheus and Grafana observability
• Familiarity with the storage hardware supply chain: drive qualification, firmware lifecycle, enclosure and backplane design, spares and RMA economics
• Experience building or running storage to meet data residency, sovereignty or compliance requirements across jurisdictions

Benefits

• A rare mandate: design the storage layer of an AI supercomputer from the ground up, at a company whose business depends on doing it better than anyone else, with the hardware budget and backing to match
• Meaningful stock options
• We're a remote-first collective, meeting in person twice a year to plan, brainstorm and celebrate wins. We keep core hours for collaboration and calls; outside those, you set your own schedule
• Our release cycles are fast and intense, and they're followed by real downtime. We expect you to take it
• Generous paid time off, including vacation, sick days and public holidays
• Paid maternity, paternity and caregiver leave
• Twice-yearly company retreats in inspiring locations