Senior System Software Engineer - LocalAI

NVIDIA Gruppe; Pune District; On-site; INR 300,000 - 550,000; Full time

Job summary

NVIDIA Gruppe in Pune, India, is seeking engineers to advance AI inference on RTX and DGX platforms. You will collaborate with software, research, and product teams to align technical priorities and drive ecosystem growth, while building high-performance local inference stacks across diverse hardware.

You will design modern runtimes, optimize models and data pipelines, and push for production-ready backends through rigorous debugging and performance analysis, leveraging tools like Llama.cpp,

Qualifications

• 5+ years of experience in Computer Science, Software Engineering, Mathematics, or related field, or equivalent experience.
• Excellent C++ programming and debugging skills with strong data structures and algorithms.
• Experience developing and optimizing AI inference pipelines using ML/DL frameworks such as Llama.cpp, vLLM, PyTorch, Windows ML, DXCGC, and TensorRT.
• Deep understanding of inference backends and runtime internals including scheduling, memory management, KV-cache, graph execution, quantization, and hardware-aware optimization.

Responsibilities

• Partner with NVIDIA’s software, research, architecture, and product teams to align technical requirements and strategic priorities for the AI ecosystem on RTX and DGX PCs.
• Build and optimize the local AI inference stack for RTX, RTX Pro, and DGX GPUs with focus on performance, stability, and scalability across diverse hardware architectures.
• Design and develop modern inference runtimes and execution stacks using frameworks such as llama.cpp, vLLM, PyTorch, WinML, DXCGC, and TensorRT for LLM, vision-language, TTS, ASR, and diffusion AI workloads.
• Perform end-to-end optimization of AI models, data pipelines, and inference runtimes to maximize performance on current and next-gen GPU architectures; apply quantization, pruning, sparsity, and distillation for efficient deployment.
• Conduct system-level debugging, performance tuning, and analysis; develop infrastructure for performance and accuracy sweeps; establish guidelines for production readiness of new models and backends.

Skills

C++

Algorithms

ML/DL frameworks

Inference pipelines

Debugging

Performance

Education

Bachelor’s/Master’s/PhD in CS or related

Tools

llama.cpp

vLLM

PyTorch

Windows ML

DXCGC

TensorRT

Job description

What You’ll Be Doing:

• Partner with NVIDIA’s software, research, architecture, and product teams to align technical requirements and strategic priorities, fostering the AI ecosystem on RTX and DGX PCs.
• Build and optimize the local AI inference stack for RTX, RTX Pro, and DGX GPUs, with a focus on performance, stability, and scalability across diverse hardware architectures.
• Design and develop modern inference runtimes and execution stacks using frameworks such as llama.cpp, vLLM, PyTorch, WinML, DXCGC, and TensorRT-RTX, supporting LLM, vision-language, TTS, ASR, and diffusion-based AI workloads.
• Perform end-to-end optimization of AI models, data pipelines, and inference runtimes to maximize performance on current and next-generation GPU architectures. Apply model optimization techniques, including quantization, pruning, sparsity, and distillation, to enable efficient deployment of large models on local and edge devices.
• Conduct system-level debugging, performance tuning, and performance-accuracy trade-off analysis; develop infrastructure for performance and accuracy sweeps; analyse results to identify gaps and drive fixes; and establish engineering guidelines to accelerate bring-up and ensure production readiness of new models and inference backends.

What we need to see:

• 5+ Years of experience with Bachelor’s, Master’s, or PhD in Computer Science, Software Engineering, Mathematics, or a related field, or equivalent experience.
• Excellent C++ programming and debugging skills, with a strong foundation in data structures, algorithms, and machine learning.
• Proven experience developing and optimizing AI inference pipelines and applications using ML/DL frameworks such as Llama.cpp, vLLM, PyTorch, Windows ML, DXCGC, and TensorRT.
• Deep understanding of inference backends and runtime internals, including scheduling, memory management, KV-cache behaviour, graph execution, quantization, and hardware-aware optimization.
• Strong analytical and problem-solving skills, with the ability to manage multiple priorities effectively in a fast-paced environment.
• Excellent written and verbal communication skills, enabling effective collaboration across engineering teams and management.

Ways to stand out from the crowd:

• Understanding of modern machine learning, deep neural network, and generative AI techniques, with relevant contributions to major open-source projects.
• Consistent track record of delivering end-to-end products in multinational companies with geographically distributed teams.
• Proficiency in low-level system and GPU programming, CUDA, and the development of high-performance systems.
• Contributions to open-source inference runtimes, model tooling, or performance infrastructure.
• Hands-on experience building applications using frameworks and APIs such as llama.cpp, PyTorch, TensorRT, Vulkan, DirectX, and vLLM.

We're a top employer recognized for innovation, growth, and a commitment to diversity as an equal-opportunity workplace. We offer competitive salaries, a generous benefits package, and the opportunity to work alongside some of the technology industry's most talented and forward-thinking professionals. As our engineering teams continue to grow rapidly, we're looking for creative, self-driven engineers with a passion for technology to join us.