DL Performance Software Engineer - LLM Inference

NVIDIA; Toronto; On-site; CAD 135,000 - 220,000; Full time

Benefits offered by this job

Equity

Benefits

Job summary

NVIDIA is seeking highly skilled software engineers to design and optimize AI inference systems that scale to large models with high efficiency. You will work across inference performance, kernels, training, large-scale serving, and research teams to push the frontier of accelerated AI computing.

Your work will include contributing features to vLLM, profiling inference frameworks, and developing novel runtime optimizations for serving infrastructure.

Qualifications

• Bachelor’s, Master’s, or PhD in CS/CE/SE.
• 5+ years of industry software engineering or equivalent research experience.
• Strong programming skills in Python and one of C/C++, Go, or Rust; solid CS fundamentals.
• Knowledgeable about performance engineering in ML frameworks (e.g., PyTorch) and inference engines (e.g., vLLM).
• Familiarity with GPU programming and performance: CUDA, memory hierarchy, profiling tools.

Responsibilities

• Contribute features to vLLM enabling latest NVIDIA GPU hardware features and serving runtime algorithms.
• Profile and optimize the inference framework with methods like speculative decoding and 5D parallelism.
• Architect novel frameworks and runtime optimizations for inference infrastructure, benchmarking, and kernels.
• Conduct and publish original research advancing ML systems performance and production-grade integration.
• Develop, optimize, and benchmark GPU kernels using fusion, autotuning, and memory/layout optimization.

Skills

Python

C/C++

Go

Rust

Algorithms & data structures

Operating systems

Parallel programming

Distributed systems

Deep learning theory

PyTorch

Inference engines

CUDA

Profiling tools

Education

Bachelor's degree in CS/CE/SE

Master's or PhD in CS/CE/SE

Tools

vLLM

SGLang

Nsight Systems/Compute

CUDA

Triton

Job description

We are seeking highly skilled and motivated software engineers to join us and build AI inference systems that serve large-scale models with extreme efficiency. You’ll architect and implement high-performance inference software, optimize GPU kernels, drive industry benchmarks, and work with state-of-the-art research techniques to improve serving efficiency. You’ll collaborate across inference performance, kernels, training, large-scale serving, and research teams to push the frontier of accelerated computing for AI.

What You’ll Be Doing

• Contribute features to vLLM that empower the newest models with the latest NVIDIA GPU hardware features and serving runtime algorithms.
• Profile and optimize the inference framework (vLLM) with methods like speculative decoding, 5D Parallelism, and prefill‑decode disaggregation.
• Architect novel frameworks and runtime optimizations for inference infrastructure, benchmarking, and kernels.
• Conduct and publish original research that advances the Pareto frontier in ML Systems; survey recent publications and find a way to integrate research ideas and prototypes into production‑grade, open‑source software.
• Develop, optimize, and benchmark GPU kernels (both hand‑tuned and compiler‑generated) using techniques such as fusion, autotuning, and memory/layout optimization.

What We Need To See

• Bachelor’s, Master’s, or PhD degree in Computer Science (CS), Computer Engineering (CE) or Software Engineering (SE).
• 5+ years of industry experience in software engineering or equivalent research experience.
• Strong programming skills in Python and one of C/C++, Go, or Rust. Solid CS fundamentals: algorithms & data structures, operating systems, computer architecture, parallel programming, software engineering, distributed systems, deep learning theories.
• Knowledgeable and passionate about performance engineering in ML frameworks (e.g., PyTorch) and inference engines (e.g., vLLM and SGLang).
• Familiarity with GPU programming and performance: CUDA, memory hierarchy, streams, NCCL; proficiency with profiling/debug tools (e.g., Nsight Systems/Compute).
• Excellent debugging, problem-solving, and communication skills; ability to excel in a fast‑paced, multi‑functional setting.

Ways to Stand out from the Crowd

• Experience developing major features and optimizations for LLM inference engines (e.g., vLLM, SGLang).
• Hands‑on work with LLM inference and training runtimes (deploying LLMs to production, large‑scale LLM pre‑training and RL), ML compilers and DSLs (e.g., Triton, CuTe, MLIR/LLVM, XLA), GPU libraries (e.g., CUTLASS) and features (e.g., CUDA Graph, Tensor Cores).
• Experience with speculative decoding training and runtime features: tree‑structured drafting, parallel drafting, diffusion LLMs, DFlash, EAGLE.
• Contributions to open‑source projects and/or publications; please include links to GitHub pull requests, published papers and artifacts.
• At NVIDIA, we believe artificial intelligence (AI) will fundamentally transform how people live and work. Our mission is to advance AI research and development to create groundbreaking technologies that enable anyone to harness the power of AI and benefit from its potential.

Our team consists of experts in AI, systems and performance optimization. Our leadership includes world‑renowned experts in AI systems who have received multiple academic and industry research awards. If you’re excited to build systems, kernels, and tools that make large-scale AI faster, more efficient, and easier to deploy, we’d love to hear from you.

Your base salary will be determined based on your location, experience, and the pay of employees in similar positions. The base salary range is 135,000 CAD - 185,000 CAD for Level 3, and 170,000 CAD - 220,000 CAD for Level 4.

You will also be eligible for equity and benefits.

Applications for this job will be accepted at least until August 10, 2026.

This posting is for an existing vacancy.

NVIDIA uses AI tools in its recruiting processes.