Senior AI Inference Orchestration Lead

Renice AI, Inc.; Mountain View (CA); On-site; USD 180,000 - 280,000; Full time

Job summary

Renice AI in Mountain View, CA is seeking an AI Inference Routing & Scheduling Lead to design and operate the control plane for large-scale, multi-tenant LLM serving. You will translate complex system requirements into practical production policies and own the end-to-end routing, scheduling, and resilience of the service.

You will work across teams—machine learning, infrastructure, Kubernetes, networking, reliability, and security—to implement online algorithms, telemetry contracts, and robust

Qualifications

• Experience owning or building production routing, scheduling, load-balancing, or resource-management systems at large scale.
• Deep knowledge of distributed systems and at least one of online algorithms, queueing theory, optimization, control, or operations research.
• Understanding of LLM inference behavior, including prefill and decode asymmetry, continuous batching, KV-cache growth and reuse, and tail-latency tradeoffs.
• Strong systems programming skills in Go, Python, C++, Rust, or a comparable language, with experience debugging production performance and reliability.
• Ability to turn ambiguous, open-ended objectives into measurable models, experiments, and production policies.
• Clear communication and the ability to influence internal teams and external infrastructure partners.

Responsibilities

• Build and own the production routing and scheduling control plane for multi-tenant LLM inference at data center scale.
• Design online algorithms that balance time to first token, inter-token latency, throughput, KV-cache reuse, accelerator utilization, cost, and SLA risk.
• Develop quantitative policies for admission control, overload protection, load shedding, cache-aware routing, priority, fairness, batching, and tenant resource allocation.
• Define an engine-neutral telemetry and service-profile contract for vLLM, SGLang, and future serving runtimes.
• Own routing correctness and resilience, including stale endpoint detection, draining, failover, recovery, and exactly-once streaming semantics.
• Build trace replay, simulation, benchmarking, and shadow-decision systems to validate policies before production rollout.
• Partner with machine learning, infrastructure, Kubernetes, networking, reliability, and security teams to translate service requirements into dependable systems.
• Lead and grow a small team of two to three engineers, set technical direction, and maintain high standards for systems rigor and operational quality.

Skills

Distributed systems

Online algorithms

Queueing theory

Go/Python/C++/Rust

Scheduling

Communication

Tools

Kubernetes

LLM serving tech

Ray Serve

Triton

Golang

Python

C++

Job description

Renice AI in Mountain View, CA is seeking an AI Inference Routing & Scheduling Lead to design and operate the control plane for large-scale, multi-tenant LLM serving. You will translate complex system requirements into practical production policies and own the end-to-end routing, scheduling, and resilience of the service.

You will work across teams—machine learning, infrastructure, Kubernetes, networking, reliability, and security—to implement online algorithms, telemetry contracts, and robust