Senior System Software Engineer - AI Inference Platform

NVIDIA; Santa Clara (CA); Hybrid; USD 272,000 - 431,250; Full time

Benefits offered by this job

Equity

Benefits

Hybrid work

Job summary

NVIDIA is seeking a Senior System Software Engineer to join the Dynamo team, building a Generative AI inference platform on GPU-accelerated hardware. You will contribute to open source components that power inference of trained AI models and advance disaggregated serving for Dynamo-supported engines and agentic workloads.

The role emphasizes high-performance software design, memory and cache optimization, and stateful, multi-turn execution across heterogeneous memory hierarchies.

Qualifications

• Masters or PhD or equivalent experience.
• 10+ years in Computer Science, Computer Engineering, or related field.
• Ability to work in a fast-paced, agile team environment.
• Excellent Rust/Python programming and software design skills, including debugging, performance analysis, and test design.
• Understanding of modern LLM API semantics, including structured outputs, tool calling, reasoning controls, token accounting, context management, and multimodal inputs.

Responsibilities

• Develop open source software to serve inference of trained AI models running on GPUs.
• Contribute to the development of disaggregated serving for Dynamo-supported inference engines and expand capabilities for agentic inference workloads.
• Innovate in inference-state management across memory and storage hierarchies to improve latency and throughput.
• Build and evolve Dynamo’s distributed inference frontend with model support and API compatibility.
• Balance and optimize robust, scalable software components for distributed inference workloads.

Skills

Rust/Python programming

LLM API semantics

10+ years experience

Agile teamwork

Open source software development

Education

Masters or PhD or equivalent

Tools

vLLM

TensorRT-LLM

SGLang

Job description

NVIDIA is seeking a Senior System Software Engineer to join the Dynamo team, building a Generative AI inference platform on GPU-accelerated hardware. You will contribute to open source components that power inference of trained AI models and advance disaggregated serving for Dynamo-supported engines and agentic workloads.

The role emphasizes high-performance software design, memory and cache optimization, and stateful, multi-turn execution across heterogeneous memory hierarchies.