Senior GPU ML Inference Engineer — Edge AI Platforms

NVIDIA; Westford (MA); On-site; USD 224,000 - 431,250; Full time

Job summary

NVIDIA is building the software stack for fast LLM inference on edge AI hardware in Westford, MA. This role focuses on evaluating open-source inference frameworks and mapping architectures to NVIDIA GPUs to maximize throughput and minimize latency.

You will own validation workflows, develop recipes, and collaborate with the community and partners to solve hardware-specific inference challenges, with equity and benefits on offer.

Qualifications

• BS, MS, or PhD in Computer Science, Computer Engineering, Electrical Engineering, or equivalent experience.
• 12+ years of software engineering with depth in GPU computing, ML systems, or high-performance inference
• Strong Python or C++ programming, software design, and software engineering skills.
• Hands-on experience with GPU kernel development or optimization (CUDA/C++, Triton, or equivalent) — you understand how thread blocks, memory hierarchy, and warp execution affect real-world performance
• Working knowledge of LLM inference internals: attention mechanisms, KV-cache management, continuous batching, quantization formats, and tensor parallelism
• Container engineering expertise: multi-architecture Docker or OCI builds, layer optimization, runtime configuration, NVIDIA Container Toolkit
• Strong analytical skills: ability to form a performance hypothesis, design an experiment, interpret results, and communicate findings clearly

Responsibilities

• Track and evaluate innovations in open-source LLM inference frameworks for NVIDIA edge AI hardware.
• Analyze model architectures and inference algorithms on NVIDIA GPUs to identify mismatches and optimization opportunities.
• Characterize multi-node inference: NCCL/RCCL, topology-aware all-reduce, and parallelism on edge clusters.
• Produce performance reports mapping hardware limits to observed throughput, latency, and utilization.
• Own model validation for new releases: architecture compatibility, recipe development, performance characterization, and publication to recipe sites.
• Develop and maintain developer-facing inference recipes; automate staleness detection and CI feedback loops.
• Engage with community and partners on model bring-up; be the technical contact for hardware-specific inference issues.

Skills

Python

C++

GPU computing

ML systems

Performance optimization

Triton

CUDA

KV-cache

Education

BS/MS/PhD in CS/CE/EE

Tools

NVIDIA Container Toolkit

Docker

OCI builds

Job description

NVIDIA is building the software stack for fast LLM inference on edge AI hardware in Westford, MA. This role focuses on evaluating open-source inference frameworks and mapping architectures to NVIDIA GPUs to maximize throughput and minimize latency.

You will own validation workflows, develop recipes, and collaborate with the community and partners to solve hardware-specific inference challenges, with equity and benefits on offer.