Engineering Manager, GPU AI Inference at Scale

NVIDIA; New York (NY); On-site; USD 184,000 - 357,000; Full time

Benefits offered by this job

Equity

Benefits

Job summary

NVIDIA is seeking an exceptional Manager, Deep Learning Inference Software, to lead a world-class engineering team advancing AI model deployment. You will shape software powering today’s most sophisticated AI systems — from large language models to multimodal generative AI — all accelerated on NVIDIA GPUs.

You will drive strategy, roadmap, and end-to-end inference pipelines, mentor engineers, and optimize performance across CUDA, Triton, CUTLASS, and multi-GPU configurations.

Qualifications

• MS/PhD or equivalent in CS/EE or related field.
• 6+ years of software development experience with 3+ years in leadership.
• Strong C/C++ design; Python a plus.
• Hands-on GPU programming (CUDA, Triton, CUTLASS) and optimization.
• Proven track record deploying DL models in production.
• Experience with Agile or collaborative software practices.

Responsibilities

• Lead, mentor, and scale a high-performing engineering team focused on DL inference and GPU-accelerated software.
• Drive strategy, roadmap, and execution of inference frameworks for Client AI.
• Partner with compiler, libraries, and research teams to deliver optimized inference pipelines across NVIDIA accelerators.
• Oversee performance tuning and optimization for large-scale models (LLMs, multimodal, generative AI).
• Guide engineers in multi-GPU communications (NIXL, NCCL, NVSHMEM) and CUDA ecosystem practices.
• Represent the team in planning discussions, aligning with broader AI and software strategies.

Skills

C/C++ software design

Python proficiency

GPU programming

Leadership / management

Agile development

Education

MS or PhD in CS/EE or related field

Tools

CUDA

Triton

CUTLASS

NCCL

NVSHMEM

Job description

NVIDIA is seeking an exceptional Manager, Deep Learning Inference Software, to lead a world-class engineering team advancing AI model deployment. You will shape software powering today’s most sophisticated AI systems — from large language models to multimodal generative AI — all accelerated on NVIDIA GPUs.

You will drive strategy, roadmap, and end-to-end inference pipelines, mentor engineers, and optimize performance across CUDA, Triton, CUTLASS, and multi-GPU configurations.