Senior Backend Engineer — Remote Inference Gateway & Routing

You.com; San Francisco (CA); Remote; USD 200,000 - 250,000; Full time

Job summary

You.com is hiring for a remote role in the United States. You will own and scale the inference gateway, spanning model routing, cost attribution, and enforcement on the request path.

You’ll work between model serving and product to optimize latency, cost, and reliability, with preference for candidates in SF Bay Area or NYC but open to strong remote applicants across the US. The position emphasizes building a scalable gateway with a single-digit-millisecond budget, strong Python/Kubernetes

Qualifications

• Hands-on production experience with model serving on stacks such as vLLM, SGLang (or similar platforms).
• Proven ability to tune serving stacks against live traffic to improve p99.
• Fluent with provider APIs: tokenization, prompt-caching, usage reporting and pricing.
• Experience building or operating a proxy, gateway, or data plane on production traffic.
• Comfort with streaming responses, partial state, and mid-stream failures.
• Strong backend generalist skills in Python and Kubernetes.
• Trajectory and recent growth are valued over years of experience.

Responsibilities

• Own the inference gateway that routes every model call, including selection, fallback and caps.
• Build an intelligence layer to autonomously pick the right model for each task turn.
• Make routing decisions considering prefix-cache, batching, and time-to-first-token.
• Implement cost attribution and metering across providers, with accurate billing.
• Enforce request-path controls that degrade gracefully without service disruption.
• Ship changes in shadow mode first with measurement to prove correctness.

Skills

model-serving-experience

traffic-tuning

provider-APIs

proxy-gateway

streaming-responses

backend-generalist

growth-trajectory

Tools

vLLM

SGLang

Kubernetes

Python

Job description

You.com is hiring for a remote role in the United States. You will own and scale the inference gateway, spanning model routing, cost attribution, and enforcement on the request path.

You’ll work between model serving and product to optimize latency, cost, and reliability, with preference for candidates in SF Bay Area or NYC but open to strong remote applicants across the US. The position emphasizes building a scalable gateway with a single-digit-millisecond budget, strong Python/Kubernetes