Senior Software Engineer, Quantized Inference

NVIDIA Corporation; Santa Clara (CA); On-site; USD 152,000 - 288,000; Full time

Benefits offered by this job

Equity

Benefits

Job summary

NVIDIA Corporation in Santa Clara, CA seeks a Senior Software Engineer specializing in Quantized Inference to speed up LLM deployment. You will implement quantized and sparse recipes in inference engines, optimize export pipelines, and build benchmarks for throughput and interactivity across Megatron-LM, vLLM, and related tools.

Key work includes writing Triton kernels, integrating quantize/dequantize paths, and collaborating with cross-team inference groups to push performance.

Qualifications

• Proficient in Python; familiarity with C++
• Strong software engineering fundamentals: concise, well-tested code; fluent with AI-assisted tooling
• Experience with ML accelerators with a basic understanding of how certain ML layers affect execution time
• Familiarity with PyTorch internals (custom ops, autograd, export) or equivalent framework
• MS/PhD in Computer Science or related field, or equivalent experience
• 4+ years in a relevant software engineering role

Responsibilities

• Implement quantized and sparse recipes in inference engines (vLLM, TRT-LLM, SGLang)
• Own model export pipelines (ModelOpt, Megatron-LM <-> HuggingFace)
• Build prototypes and benchmarking harnesses to evaluate recipe throughput/interactivity
• Develop data analysis tooling and visualizations for numerics debugging
• Improve developer productivity across the team: CI, build systems, training infrastructure, pipeline friction
• Participate in code reviews and incorporate feedback

Skills

Python

C++

PyTorch internals

Triton kernels

AI-assisted tooling

Education

MS/PhD in Computer Science or related field

Tools

PyTorch

vLLM

TRT-LLM

SGLang

HuggingFace ModelOpt

Job description

We are now looking for a Senior Software Engineer for Quantized Inference! NVIDIA is seeking software engineers to accelerate the discovery and deployment of efficient inference recipes for LLMs. A recipe defines which operators are transformed into low-precision or sparsified variants — unlocking throughput and latency gains without regressing accuracy or verbosity. Recipes may incorporate techniques such as rotations, block scaling to attenuate outlier impact, or improved calibration data drawn from SFT/RL pipelines.Each new recipe demands corresponding kernel and model-level implementations in inference engines (vLLM, TRT-LLM, SGLang). The candidate will translate recipe specifications into functionally correct, performant code, e.g., writing Triton kernels, inserting quantize/dequantize nodes into prefill and decode paths, and ensuring per-expert scaling in MoE layers is handled correctly. From there, the candidate will collaborate with partner inference teams to further optimize throughput and interactivity on target workloads. This work is a core component of our productization effort across Megatron-LM, ModelOpt, and vLLM.What you'll be doing:Implement quantized and sparse recipes in inference engines (vLLM, TRT-LLM, SGLang)Own model export pipelines (ModelOpt, Megatron-LM <-> HuggingFace), ensuring quantized checkpoints serialize correctly for downstream servingBuild prototypes and benchmarking harnesses to evaluate recipe throughput/interactivity before full optimizationDevelop data analysis tooling and visualizations for numerics debuggingImprove developer productivity across the team: CI, build systems, training infrastructure, pipeline frictionParticipate in code reviews and incorporate feedbackWhat we need to see:Proficient in Python; familiarity with C++Strong software engineering fundamentals: concise, well-tested code; fluent with AI-assisted toolingExperience with ML accelerators with a basic understanding of how certain ML layers affect execution timeFamiliarity with PyTorch internals (custom ops, autograd, export) or equivalent frameworkExperience reading, modifying, or contributing to a large open-source codebaseMS/PhD in Computer Science or related field, or equivalent experience.4+ years in a relevant software engineering roleDemonstrated ability to move fast with ambiguous requirements, with strong written and verbal communicationWays to stand out from the crowd:Experience contributing to inference serving frameworks (vLLM, TRT-LLM, SGLang) or Triton kernel developmentTrack record of debugging numerical issues across mixed-precision boundariesDeep experience with model compression techniques: PTQ, QAT, structured/unstructured sparsityYour base salary will be determined based on your location, experience, and the pay of employees in similar positions. The base salary range is 152,000 USD - 241,500 USD for Level 3, and 184,000 USD - 287,500 USD for Level 4.You will also be eligible for equity and benefits.Applications for this job will be accepted at least until July 26, 2026.This posting is for an existing vacancy.NVIDIA uses AI tools in its recruiting processes.NVIDIA is committed to fostering an inclusive work environment and proud to be an equal opportunity employer. As we highly value diversity in our current and future employees, we do not discriminate (including in our hiring and promotion practices) on the basis of race, religion, color, national origin, gender, gender expression, sexual orientation, age, marital status, veteran status, disability status or any other characteristic protected by law.