On-Site Inference Systems Engineer — Rust/LLM Runtime

Confidential; California (MO); On-site; USD 150,000 - 210,000; Full time

Job summary

Confidential in Palo Alto is hiring a Member of Technical Staff, Inference Systems to build a from-scratch Rust runtime for LLM serving. You’ll own batching, scheduling, KV cache management and the full serving stack, shaping architecture with the founding team.

You’ll work on latency-sensitive, high-throughput systems on-site five days a week, with the opportunity to influence design decisions and optimize cost per token in a cutting-edge AI platform.

Qualifications

• 2-10 years of experience as a backend, systems, or distributed systems engineer.
• Deep familiarity with inference internals: attention, KV cache, batching, scheduling.
• Hands-on time inside engines like vLLM, SGLang, or TensorRT-LLM, or experience building LLM serving infrastructure.
• Strong systems programming skills in Rust, C++, or similar, and a genuine willingness to work in Rust day to day.
• Experience profiling and optimizing performance‑critical systems.
• Comfort owning an entire stack rather than a narrow slice.

Responsibilities

• Build the serving runtime from scratch in Rust, design KV cache management and prefix caching that cut latency and cost per token.
• Scale serving across multi-GPU and multi-node setups.
• Profile and benchmark the full inference pipeline.
• Work directly with the founding team on the architecture that defines the platform.

Skills

Backend systems

Distributed systems

Rust

Performance profiling

Tools

vLLM

SGLang

TensorRT-LLM

CUDA

Triton

NCCL

Job description

Confidential in Palo Alto is hiring a Member of Technical Staff, Inference Systems to build a from-scratch Rust runtime for LLM serving. You’ll own batching, scheduling, KV cache management and the full serving stack, shaping architecture with the founding team.

You’ll work on latency-sensitive, high-throughput systems on-site five days a week, with the opportunity to influence design decisions and optimize cost per token in a cutting-edge AI platform.