Member of Technical Staff, Inference

OpenRelay, Inc.; San Francisco (CA); Hybride; USD 180 000 - 240 000; Plein temps

Résumé du poste

OpenRelay, Inc. in San Francisco/Seattle is seeking a high-autonomy engineer to own production LLM serving, from problem discovery to production, collaborating directly with the founders to shape technical direction.

You will optimize performance, cost, and reliability of models, bring new open-weight models to production, and tune GPU serving with batching, caching, and quantization. You will design routing across replicas andpaces, and push measurable wins in throughput and latency.

Qualifications

• 3+ years of industry experience in production LLM serving.
• Hands-on with vLLM, SGLang, and NVIDIA Dynamo.
• Strong Python and PyTorch, able to read/modify large codebases.
• Track record of measurable throughput, latency, or cost improvements.
• Own end-to-end projects with high autonomy.

Responsabilités

• Own the performance, cost, and reliability of every model we serve
• Bring new open-weight models to production quickly, including ones that current engine releases do not yet support
• Tune serving for each model and GPU: parallelism, batching, caching, and quantization
• Design how requests are routed and scheduled across replicas and nodes
• Profile and benchmark under realistic traffic, fix what the numbers point to, and send general fixes upstream
• Decide what to build next, and say no to work that does not move the numbers
• 3+ years of industry experience, with production LLM serving a real part of it
• Hands-on production experience with vLLM, SGLang, and NVIDIA Dynamo
• Strong Python and PyTorch, and the ability to read and change large codebases you did not write
• A track record of measurable wins in throughput, latency, or cost, with numbers you can defend
• You do your best work with high autonomy: you scope your own projects, ship them, and own the outcome

Connaissances

Python

PyTorch

LLM Serving

vLLM

SGLang

NVIDIA Dynamo

Large Codebases

Autonomy

Outils

CUDA

Triton

ROCm

Description du poste

San Francisco or Seattle · Full time · GPU compute and inference platform (YC S26)

We are a small team backed by Y Combinator. Everyone here owns large, important systems that are live in production.

About the role

This is a high-autonomy role. There is no ticket queue. You will find the most important problem, decide how to solve it, ship it to production, and own the result. You will work directly with the founders and help set our technical direction.

Responsibilities

• Own the performance, cost, and reliability of every model we serve
• Bring new open-weight models to production quickly, including ones that current engine releases do not yet support
• Tune serving for each model and GPU: parallelism, batching, caching, and quantization
• Design how requests are routed and scheduled across replicas and nodes
• Profile and benchmark under realistic traffic, fix what the numbers point to, and send general fixes upstream
• Decide what to build next, and say no to work that does not move the numbers
• 3+ years of industry experience, with production LLM serving a real part of it
• Hands-on production experience with vLLM, SGLang, and NVIDIA Dynamo
• Strong Python and PyTorch, and the ability to read and change large codebases you did not write
• A track record of measurable wins in throughput, latency, or cost, with numbers you can defend
• You do your best work with high autonomy: you scope your own projects, ship them, and own the outcome

Strong candidates may also have

• Contributions to an open-source inference engine
• GPU kernel or low-level performance work (CUDA, ROCm, or Triton)
• Experience serving mixture-of-experts models or running multi-node inference
• Experience at an early-stage startup

Representative projects

• Get a newly released open model serving in production within days of its release, then make it fast
• Cut cost per token on a high-traffic model by retuning parallelism, batching, and caching, and prove it with benchmarks,
• Route repeat prompts to the replicas that already hold their KV cache
• Trace a tail-latency regression from the request down to a scheduler or kernel, and fix it

Logistics

• Location: San Francisco or Seattle
• Compensation: competitive salary and equity