Senior Backend Engineer - AI Inference Platform

Mixpeek; San Francisco (CA); Remote; USD 200,000 - 250,000; Full time

Benefits offered by this job

Office hubs SF/NYC

Flexible PTO

Health insurance

Function Health

Parental leave 12w

401k 3% match

WFH stipend

Tech stipend

Wellness allowance

AI research collaboration

Job summary

You.com is seeking a backend generalist to own the inference gateway that routes all model calls. You will design and optimize provider/model selection, caching behavior, and metering, ensuring reliable, cost-aware routing at scale.

The role emphasizes building aproxy/gateway on the critical data path, with a focus on streaming responses and graceful degradation in production traffic.

Qualifications

• Hands-on production experience with model serving stacks like vLLM or SGLang.
• Experience tuning a serving stack against live traffic and understanding p99.
• Fluency with provider APIs, tokenization, prompt-caching, usage reporting, and pricing.

Responsibilities

• Own the inference gateway that every model call routes through, including provider and model selection, fallback, and caps
• Build intelligence layer to autonomously pick the right model for the right turn in the task
• Make routing decisions that account for prefix-cache behavior, batching, and time to first token
• Build cost attribution and metering across providers, including cached-token accounting, accurate enough to bill and enforce on
• Build request-path enforcement that degrades gracefully and never takes down a customer's inference call
• Ship inference-time changes in shadow mode first, with the measurement to prove they work

Skills

Python

Kubernetes

vLLM

SGLang

Gateway proxy

Tools

vLLM

SGLang

Job description

You.com is seeking a backend generalist to own the inference gateway that routes all model calls. You will design and optimize provider/model selection, caching behavior, and metering, ensuring reliable, cost-aware routing at scale.

The role emphasizes building aproxy/gateway on the critical data path, with a focus on streaming responses and graceful degradation in production traffic.