Staff Engineer, LLM Inference & Production

OpenRelay, Inc.; San Francisco (CA); Hybrid; USD 180,000 - 240,000; Full time

Job summary

OpenRelay, Inc. in San Francisco/Seattle is seeking a high-autonomy engineer to own production LLM serving, from problem discovery to production, collaborating directly with the founders to shape technical direction.

You will optimize performance, cost, and reliability of models, bring new open-weight models to production, and tune GPU serving with batching, caching, and quantization. You will design routing across replicas andpaces, and push measurable wins in throughput and latency.

Qualifications

• 3+ years of industry experience in production LLM serving.
• Hands-on with vLLM, SGLang, and NVIDIA Dynamo.
• Strong Python and PyTorch, able to read/modify large codebases.
• Track record of measurable throughput, latency, or cost improvements.
• Own end-to-end projects with high autonomy.

Responsibilities

• Own the performance, cost, and reliability of every model we serve
• Bring new open-weight models to production quickly, including ones that current engine releases do not yet support
• Tune serving for each model and GPU: parallelism, batching, caching, and quantization
• Design how requests are routed and scheduled across replicas and nodes
• Profile and benchmark under realistic traffic, fix what the numbers point to, and send general fixes upstream
• Decide what to build next, and say no to work that does not move the numbers
• 3+ years of industry experience, with production LLM serving a real part of it
• Hands-on production experience with vLLM, SGLang, and NVIDIA Dynamo
• Strong Python and PyTorch, and the ability to read and change large codebases you did not write
• A track record of measurable wins in throughput, latency, or cost, with numbers you can defend
• You do your best work with high autonomy: you scope your own projects, ship them, and own the outcome

Skills

Python

PyTorch

LLM Serving

vLLM

SGLang

NVIDIA Dynamo

Large Codebases

Autonomy

Tools

CUDA

Triton

ROCm

Job description

OpenRelay, Inc. in San Francisco/Seattle is seeking a high-autonomy engineer to own production LLM serving, from problem discovery to production, collaborating directly with the founders to shape technical direction.

You will optimize performance, cost, and reliability of models, bring new open-weight models to production, and tune GPU serving with batching, caching, and quantization. You will design routing across replicas andpaces, and push measurable wins in throughput and latency.