Senior AI Inference Platform Lead

NVIDIA; Santa Clara (CA); On-site; USD 208,000 - 328,000; Full time

Benefits offered by this job

Equity

Benefits package

Job summary

NVIDIA seeks a highly technical Product Manager to own AI inference optimization across the stack, from model representation to serving. You will translate deep optimization techniques into products that scale for a broad range of customers and deployment topologies.

You will collaborate on TensorRT-LLM, vLLM, SGLang, and NVIDIA Dynamo, define performance benchmarks, and drive releases with quality metrics and regression rigor while incorporating customer feedback into the roadmap.

Qualifications

• 12+ years in product management at a technology company, or comparable time as a founder, engineering lead, or technical product owner.
• Depth in AI inference optimization: KV caching and reuse, quantization, speculative decoding, disaggregated serving.
• Familiarity with the inference and orchestration frameworks customers use: TensorRT-LLM, vLLM, SGLang, NVIDIA Dynamo, and the surrounding serving and scheduling ecosystem.
• Proven track record of working independently — you can take an ambiguous problem space, define the strategy, and drive it to a shipped outcome without waiting to be told what to do next.
• Operational experience running a live product: release management, quality and regression rigor, customer issues, and support processes.
• Skill at translating low-level capability into business value — lower TCO, faster response, better GPU utilization — for engineers and executives alike.

Responsibilities

• Own the inference performance roadmap. Set direction across the stack: how models are represented, how memory and state are managed, how requests are scheduled and served, and how tokens get generated.
• Build platforms, not one-offs. Deliver capabilities that generalize across model families, deployment topologies, and customer sizes.
• Agentic and Multi-Turn Workloads: Define the performance strategy for agentic applications, where long-running sessions, tool-call stalls, and unpredictable output lengths break the assumptions built into single-turn serving.
• Framework & Ecosystem Strategy: Define how our optimizations land across TensorRT-LLM, vLLM, SGLang, and NVIDIA Dynamo. Partner with open-source communities and internal engineering teams so customers get great performance on NVIDIA hardware.
• Benchmarking & Performance Claims: Own how performance is measured, published, and reproduced. Define the benchmark methodology, the metrics that matter (TTFT, ITL, throughput per GPU, cost per million tokens), and the guardrails that keep our numbers credible.
• Run the product day to day. Own release readiness, quality bars, regression tracking, customer blocking issues, and the feedback loop from production deployments back into the roadmap.

Skills

AI inference optimization

KV caching & reuse

Quantization

Speculative decoding

Disaggregated serving

Model/framework familiarity

TensorRT-LLM

vLLM

SGLang

NVIDIA Dynamo

Release management

Regression testing

Customer issue handling

Business value translation

Education

BS/MS/PhD in Computer Science or Computer Engineering

Job description

NVIDIA seeks a highly technical Product Manager to own AI inference optimization across the stack, from model representation to serving. You will translate deep optimization techniques into products that scale for a broad range of customers and deployment topologies.

You will collaborate on TensorRT-LLM, vLLM, SGLang, and NVIDIA Dynamo, define performance benchmarks, and drive releases with quality metrics and regression rigor while incorporating customer feedback into the roadmap.