Product Manager, AI Inference Platform & Performance

Baseten; New York (NY); Remote; USD 140,000 - 210,000; Full time

Benefits offered by this job

Competitive compensation

Meaningful equity

Comprehensive US medical/dental/vision

Flexible PTO including Winter Break

Parental leave

Fertility and family-building stipend

401(k) company-facilitated

Learning opportunities

Job summary

Baseten is building a cloud platform for AI inference. We seek a deeply technical Product Manager to translate advances in model serving and performance into customer-ready capabilities.

You will bridge systems engineering and production workloads, collaborating with inference engineers and researchers to ship features that customers can adopt confidently. You’ll work closely with researchers and engineers, prioritizing optimizations across hardware, model configs, autoscaling, and routing to

Qualifications

• Experience building and shipping AI/ML products or infrastructure with performance, scalability, or production focus.
• A technical background in software engineering, ML engineering, or equivalent hands-on experience.
• Strong understanding of model serving architectures, GPU economics, latency, throughput, quality, and cost tradeoffs.
• Experience working with researchers or infrastructure engineers to bring complex capabilities into production.
• Rigorous benchmarking, experimentation, and performance measurement approach.
• Strong product judgment and ability to translate deep tech into customer value.

Responsibilities

• Work with inference engineering and research to bring new performance capabilities to customers.
• Translate advances in batching, caching, quantization, and serving architectures into product features.
• Define benchmarks and evaluations showing latency, throughput, quality, and cost improvements.
• Understand customer workloads and prioritize optimizations with the greatest impact.
• Make performance features easier to configure, evaluate, and adopt in production.
• Partner with engineering on release readiness, compatibility, and regression testing.

Skills

AI/ML product experience

Performance/scale focus

Benchmarking

Cross-functional leadership

Customer-facing

Education

Bachelor's in CS/EE/related

Tools

PyTorch

vLLM

TensorRT-LLM

SGLang

NVIDIA Dynamo

Job description

Baseten is building a cloud platform for AI inference. We seek a deeply technical Product Manager to translate advances in model serving and performance into customer-ready capabilities.

You will bridge systems engineering and production workloads, collaborating with inference engineers and researchers to ship features that customers can adopt confidently. You’ll work closely with researchers and engineers, prioritizing optimizations across hardware, model configs, autoscaling, and routing to