Senior Backend Engineer (Inference Platform)

you.com; New York (NY); Remote; USD 150,000 - 210,000; Full time

Benefits offered by this job

Remote-friendly culture

Competitive compensation

Job summary

you.com is seeking an experienced backend engineer to own and scale the inference gateway that routes every model call. You will manage provider/model selection, fallback, and enforcement within a sub-millisecond budget while building an intelligence layer for selecting the right model.

This role emphasizes cost attribution, metering, and robust, scalable routing across hosted providers and internal models.

Qualifications

• Hands-on production experience with model serving stacks on relevant platforms.
• Strong backend generalist with Python and Kubernetes experience.
• Familiarity with tokenization, prompt-caching, and usage reporting.

Responsibilities

• Own and scale the inference gateway between model serving and products.
• Build routing, fallback, and model/provider selection with low latency.
• Implement cost attribution, metering, and billing for cross-provider traffic.
• Ensure graceful degradation and shadow-mode deployment for changes.

Skills

Python

Kubernetes

Backend engineering

Model serving

Latency optimization

Caching

Billing/cost optimization

Provider APIs

Streaming responses

Tools

vLLM

SGLang

Job description

• Every model call from our products and customers passes through one layer: our inference gateway. It allows users to select a model and we intelligently route the call to optimize on cost , latency and high availability. We are now building the control plane above it, to manage and govern access, performance and cost
• You will own and scale that gateway. The work sits between model serving and product: routing and fallback across hosted providers and our own models, cache-aware routing decisions, metering accurate enough to bill on, and enforcement on the request path within a single-digit-millisecond budget
• Own the inference gateway that every model call routes through, including provider and model selection, fallback, and caps
• Build intelligence layer to autonomously pick the right model for the right turn in the task
• Make routing decisions that account for prefix-cache behavior, batching, and time to first token, not only price and availability
• Build cost attribution and metering across providers, including cached-token accounting, accurate enough to bill and enforce on
• Build request-path enforcement that degrades gracefully and never takes down a customer’s inference call
• Ship inference-time changes in shadow mode first, with the measurement to prove they work

Benefits

• A remote-first work environment with hubs in California, NYC, and Canada that offer monthly in-person gatherings
• Unlimited PTO with 11 U.S. holidays observed and a week shutdown in December to rest and recharge
• Chance to collaborate with a team at the forefront of AI research
• 12 weeks of paid paternity leave in the US; additional time off is also considered
• In-person coworking weeks 1 to 2 times a year
• 401k program, 3% match coming January 2025
• Competitive health insurance plan, where 100% of the policyholder is covered and 75% for dependents
• $1,200 / year Health & Wellness Allowance to support your personal goals
• $500 work-from-home stipend to be used up to a year of your start date

Experience building or operating a proxy, gateway, or data plane on the critical path of production traffic, with sound instincts for failing openA strong backend generalist (Python, Kubernetes). We expect you to pick up whatever you have not touched yetHands-on production experience with model serving on stacks such as vLLM, SGLang (or similar platforms), with a working understanding of continuous batching, KV and prefix caching, and the tradeoff between time to first token and throughputFluency across provider APIs: tokenization, prompt-caching semantics, and the differences in how usage and cached tokens are reported and pricedYou have tuned a serving stack against live traffic and know what moves p99 in practice, not only in theoryWe weigh trajectory and recent growth more heavily than years of experienceComfort with streaming responses and the parts that get awkward: partial state, mid-stream failures, and work that can only happen once the stream closesUsage metering, billing, or cost optimization at scaleExperience at an inference provider or GPU cloud, or contributions to an open-source serving stackFine-tuning and evaluating models in production