Senior Software Engineer, AI Inference Systems

NVIDIA Gruppe; Santa Clara (CA); On-site; USD 184,000 - 356,500; Full time

Benefits offered by this job

Equity opportunities

Comprehensive benefits package

Job summary

NVIDIA Gruppe is looking for skilled software engineers to develop AI inference systems that operate with high efficiency. The role involves architecting high-performance inference frameworks and optimizing GPU processes. Ideal candidates should have extensive programming experience and strong educational backgrounds in computer science or engineering.

Compensation ranges from 184,000 USD to 356,500 USD based on experience. Apply before May 2, 2026.

Qualifications

• 7+ years of experience in software engineering or equivalent.
• Strong programming skills in Python and C/C++.
• Experience with Go or Rust is a plus.

Responsibilities

• Optimize the inference framework using advanced techniques.
• Benchmark GPU kernels using optimization techniques.
• Define and build inference benchmarking tools.

Skills

Python

C/C++

GPU programming

Performance engineering in ML frameworks

Parallel programming

Education

Bachelor’s degree in Computer Science or equivalent

Master’s degree in Computer Science/Engineering

PhD in ML Systems or related field

Tools

Docker

Kubernetes

CUDA

Nsight Systems/Compute

Job description

Position Overview

We are seeking highly skilled and motivated software engineers to join us and build AI inference systems that serve large-scale models with extreme efficiency. You’ll architect and implement high-performance inference stacks, optimize GPU kernels and compilers, drive industry benchmarks, and scale workloads across multi-GPU, multi-node, and multi-cloud environments. You’ll collaborate across inference, compiler, scheduling, and performance teams to push the frontier of accelerated computing for AI.

At NVIDIA, we believe artificial intelligence (AI) will fundamentally transform how people live and work. Our mission is to advance AI research and development to create groundbreaking technologies that enable anyone to harness the power of AI and benefit from its potential. Our team consists of experts in AI, systems and performance optimization. If you’re excited to build systems, kernels, and tools that make large-scale AI faster, more efficient, and easier to deploy, we’d love to hear from you.

Responsibilities

• Contribute features to vLLM that empower the newest models with the latest NVIDIA GPU hardware features; profile and optimize the inference framework (vLLM) with methods like speculative decoding, data/tensor/expert/pipeline-parallelism, prefill-decode disaggregation.
• Develop, optimize, and benchmark GPU kernels (hand‑tuned and compiler‑generated) using techniques such as fusion, autotuning, and memory/layout optimization; build and extend high‑level DSLs and compiler infrastructure to boost kernel developer productivity while approaching peak hardware utilization.
• Define and build inference benchmarking methodologies and tools; contribute both new benchmark and NVIDIA’s submissions to the industry‑leading MLPerf Inference benchmarking suite.
• Architect the scheduling and orchestration of containerized large‑scale inference deployments on GPU clusters across clouds.
• Conduct and publish original research that pushes the Pareto frontier for the field of ML Systems; survey recent publications and find a way to integrate research ideas and prototypes into NVIDIA’s software products.

Qualifications

• Bachelor’s degree (or equivalent experience) in Computer Science (CS), Computer Engineering (CE) or Software Engineering (SE) with 7+ years of experience; alternatively, Master’s degree in CS/CE/SE with 5+ years of experience; or PhD degree with the thesis and top‑tier publications in ML Systems, GPU architecture, or high‑performance computing.
• Strong programming skills in Python and C/C++; experience with Go or Rust is a plus; solid CS fundamentals: algorithms & data structures, operating systems, computer architecture, parallel programming, distributed systems, deep learning theories.
• Knowledgeable and passionate about performance engineering in ML frameworks (e.g., PyTorch) and inference engines (e.g., vLLM and SGLang).
• Familiarity with GPU programming and performance: CUDA, memory hierarchy, streams, NCCL; proficiency with profiling/debug tools (e.g., Nsight Systems/Compute).
• Experience with containers and orchestration (Docker, Kubernetes, Slurm); familiarity with Linux namespaces and cgroups.
• Excellent debugging, problem‑solving, and communication skills; ability to excel in a fast‑paced, multi‑functional setting.

Desired Experience

• Experience building and optimizing LLM inference engines (e.g., vLLM, SGLang).
• Hands‑on work with ML compilers and DSLs (e.g., Triton, TorchDynamo/Inductor, MLIR/LLVM, XLA), GPU libraries (e.g., CUTLASS) and features (e.g., CUDA Graph, Tensor Cores).
• Experience contributing to containerization/virtualization technologies such as containerd/CRI‑O/CRIU.
• Experience with cloud platforms (AWS/GCP/Azure), infrastructure as code, CI/CD, and production observability.
• Contributions to open‑source projects and/or publications; please include links to GitHub pull requests, published papers and artifacts.

Compensation & Benefits

Your base salary will be determined based on your location, experience, and the pay of employees in similar positions. The base salary range is 184,000 USD - 287,500 USD for Level 4, and 224,000 USD - 356,500 USD for Level 5.

You will also be eligible for equity and benefits.

Applications for this job will be accepted at least until May2,2026. This posting is for an existing vacancy.

EEO Statement

NVIDIA is committed to fostering a diverse work environment and proud to be an equal opportunity employer. As we highly value diversity in our current and future employees, we do not discriminate (including in our hiring and promotion practices) on the basis of race, religion, color, national origin, gender, gender expression, sexual orientation, age, marital status, veteran status, disability status or any other characteristic protected by law.