Manager, System Software Engineering - Local AI

NVIDIA; Pune District; On-site; INR 3,500,000 - 7,000,000; Full time

Job summary

NVIDIA is seeking a System Software Manager to lead the on-device AI software stack for RTX, RTX Pro, and DGX-class systems in Pune. You will guide the architecture, optimize runtimes, and ensure a streamlined experience for developers and end users.

You will mentor engineers, align with software, research, and product teams, and drive performance-focused development of inference runtimes across Llama.cpp, vLLM, PyTorch, WinML, DXCGC, and TensorRT.

Qualifications

• Bachelor’s, Master’s, or PhD in Computer Science, Software Engineering, Mathematics or related field.
• Proven leadership of high-performing engineering teams in systems software or AI infrastructure.
• Strong foundation in C++ development, data structures, algorithms, and ML systems.
• Extensive experience with AI inference pipelines and DL frameworks like PyTorch, Llama.cpp, vLLM, TensorRT.
• Experience with inference backends, scheduling, memory management, and hardware-aware optimization.

Responsibilities

• Lead and grow the on-device AI inference platform for RTX, RTX Pro, and DGX GPUs.
• Provide technical leadership across architecture and evolution of inference runtimes.
• Mentor engineers and foster a high-performance team culture.
• Coordinate optimization of AI models, data pipelines, and runtimes for GPU architectures.
• Drive quantization, pruning, sparsity, and distillation for local deployment.
• Establish processes for system-level debugging, performance sweeps, and production readiness.

Skills

C++ development

Debugging

Data structures & algorithms

ML systems

AI inference pipelines

TensorRT

Llama.cpp

vLLM

PyTorch

CUDA

Education

Bachelor’s/Master’s/PhD in CS/Software Eng/Math

Tools

CUDA

DirectX

Vulkan

DXCGC

TensorRT

Job description

NVIDIA has continuously reinvented itself for over two decades. The invention of the GPU in 1999 propelled the growth of PC gaming, redefined modern computer graphics, and revolutionized parallel computing. More recently, GPU deep learning has ignited modern AI, positioning NVIDIA as a leading AI computing company. There is a growing focus on delivering AI models locally, closer to the source of data. This reduces latency, improves real-time processing, and addresses privacy concerns by minimizing data transfer to centralized servers. As technology advances, client-side AI (local execution) will play a key role in crafting digital experiences.

The Local AI team is seeking a System Software Manager to lead development of an efficient on-device AI software stack. The software stack will support RTX, RTX Pro, and DGX-class systems. This role focuses on high-performance local inference, agentic workloads, low latency, efficient memory use, scalable infrastructure, practical deployment on resource-constrained platforms, and delivering a streamlined out-of-box experience for developers and end users.

What You'll Be Doing

• Lead and grow a team building the on-device AI inference platform for RTX, RTX Pro, and DGX GPUs, with accountability for execution, technical direction, delivery quality, and roadmap alignment.
• Drive cross-functional alignment with NVIDIA’s software, research, architecture, and product teams, along with industry partners and open-source communities, to build strategy and strengthen the AI ecosystem across RTX and DGX platforms.
• Provide technical leadership for the architecture and evolution of modern inference runtimes and execution stacks across frameworks such as Llama.cpp, vLLM, PyTorch, WinML, DXCGC, and TensorRT-RTX, spanning workloads including LLMs, vision-language models, TTS, ASR, and diffusion models.
• Mentor engineers, develop technical leaders, and foster a high-performance team culture centred on innovation, collaboration, and operational excellence.
• Coordinate end-to-end optimization of AI models, data pipelines, and inference runtimes to improve performance across current and next-generation GPU architectures.
• Drive adoption of model optimization techniques such as quantization, pruning, sparsity, and distillation to enable efficient deployment of large models on local and edge devices.
• Establish team processes for system-level debugging, performance optimization, and performance-accuracy trade-off analysis, including infrastructure for performance and accuracy sweeps, gap analysis, and production-readiness improvements.

What We Need To See

• 5+ overall years of industry experience and 2+ years of engineering leadership experience, combined with a Bachelor’s, Master’s, or PhD in Computer Science, Software Engineering, Mathematics, or a related field.
• Proven experience leading high-performing engineering teams in systems software, AI infrastructure, inference runtimes, or related domains.
• Strong technical foundation in C++ software development, debugging, data structures, algorithms, and machine learning systems.
• Extensive background in AI inference pipelines and Deep Learning frameworks like Llama.cpp, vLLM, PyTorch, WinML, DXCGC, and TensorRT.
• Deep understanding of inference backends and runtime internals, including scheduling, memory management, KV-cache behavior, graph execution, quantization, and hardware-aware optimization.
• Strong analytical and problem-solving skills, with the ability to balance technical depth, execution speed, and organizational priorities in a fast-paced environment.
• Excellent written and verbal communication skills, with proven ability to collaborate across engineering, product, research, and executive collaborators.

Ways To Stand Out From The Crowd

• Strong understanding of modern machine learning, deep neural networks, and generative AI, along with contributions to notable open-source projects.
• Demonstrated success building teams, setting technical vision, and scaling execution through periods of rapid growth.
• Track record of delivering end-to-end products with geographically distributed teams in multinational product organizations.
• Experience in lower-level systems or GPU programming, including CUDA and high-performance systems development.
• Contributions to open-source inference runtimes, model tooling, or performance infrastructure as well as practical experience working with frameworks and APIs including Llama.cpp, PyTorch, TensorRT, Vulkan, DirectX, and vLLM.

We're a top employer known for innovation and growth. We are an equal-opportunity employer and value diversity at our company. With competitive salaries and a generous benefits package, we are widely considered to be one of the world’s most desirable employers of technology. We have some of the most forward-thinking and hardworking people in the world working for us and, due to unprecedented growth, our best-in-class engineering teams are rapidly growing. If you're a creative and autonomous engineer with a real passion for technology, we would like to hear from you.