Inference Systems Engineer - Fast, Multi-GPU Model Serving

River AI Inc.; Palo Alto (CA); On-site; USD 200,000 - 420,000; Full time

Benefits offered by this job

Comprehensive health, dental, and vis

Unlimited PTO

Relocation assistance

Visa sponsorship

Job summary

River AI Inc. is seeking exceptional inference systems engineers to build engines that serve large models through the River API.

You will own the serving runtime, from request scheduling to distributed model execution, with a focus on latency, throughput, reliability, and cost. You will work with GPU kernel engineers, researchers, and infra engineers to bring models to production, optimize performance, and ensure model-version consistency across deployments.

Qualifications

• Bachelor’s degree in Computer Science, Computer Engineering, or equivalent practical experience.
• Experience building inference engines or performance-sensitive distributed services.
• Strong understanding of transformer inference, GPU memory, concurrency, and networking.
• Proficiency in Python and C++ or Rust.
• Strong debugging and profiling skills across models, runtimes, and services.
• A collaborative mindset and strong ownership of engineering outcomes.

Responsibilities

• Optimize inference for dense and mixture-of-experts models, including fine-tuned models and adapters.
• Improve batching, caching, and admission control to balance throughput, latency, memory use, and fairness.
• Accelerate multi-GPU execution, communication, and model loading while preserving model-version consistency.
• Improve RL sampling throughput while keeping samples and log probabilities tied to the correct model version.
• Build reliable streaming, cancellation, and recovery under failures and overload.
• Profile bottlenecks and validate improvements through reproducible performance and correctness tests.

Skills

Python

C++

Rust

Transformer inference

GPU memory

Profiling

Concurrency

Networking

Education

Bachelor’s degree in CS/CE or equivalent

Tools

CUDA

TensorRT-LLM

SGLang / vLLM

Job description

River AI Inc. is seeking exceptional inference systems engineers to build engines that serve large models through the River API.

You will own the serving runtime, from request scheduling to distributed model execution, with a focus on latency, throughput, reliability, and cost. You will work with GPU kernel engineers, researchers, and infra engineers to bring models to production, optimize performance, and ensure model-version consistency across deployments.