Inference Systems Engineer — High-Performance AI Serving

River AI; Palo Alto (CA); On-site; USD 200,000 - 420,000; Full time

Benefits offered by this job

Health insurance

Dental insurance

Vision insurance

Unlimited PTO

Relocation assistance

Visa sponsorship

Job summary

River AI in Palo Alto is seeking exceptional inference systems engineers to build the engines that serve large models through the River API. Your goal is to deliver fast, reliable inference while making efficient use of GPU compute and memory.

You will own the serving runtime from request scheduling and continuous batching to KV-cache management, distributed model execution, and checkpoint loading. Working closely with GPU kernel engineers, researchers, and infrastructure engineers, you will

Qualifications

• Bachelor’s degree in CS, CE, or equivalent practical experience.
• Experience building inference engines or performance-sensitive distributed services.
• Strong knowledge of transformer inference, GPU memory, concurrency, and networking.
• Proficiency in Python and C++ or Rust.
• Strong debugging and profiling skills across models, runtimes, and services.
• Collaborative mindset with ownership of outcomes.

Responsibilities

• Optimize inference for dense and mixture-of-experts models.
• Improve batching, caching, and admission control for throughput and latency.
• Accelerate multi-GPU execution while preserving model-version consistency.
• Improve RL sampling throughput with correct model versioning.
• Build reliable streaming, cancellation, and recovery under load.
• Profile bottlenecks and validate improvements with reproducible tests.

Skills

Python

C++

Rust

Transformer inference

GPU memory

Concurrency

Networking

Debugging

Profiling

Distributed services

Education

Bachelor's degree in CS/CE

Tools

SGLang

vLLM

TensorRT-LLM

CUDA

NVIDIA profiling tools

Job description

River AI in Palo Alto is seeking exceptional inference systems engineers to build the engines that serve large models through the River API. Your goal is to deliver fast, reliable inference while making efficient use of GPU compute and memory.

You will own the serving runtime from request scheduling and continuous batching to KV-cache management, distributed model execution, and checkpoint loading. Working closely with GPU kernel engineers, researchers, and infrastructure engineers, you will