Senior AI Systems Engineer: Distributed GPU Training

NVIDIA Corporation

United States

On-site

USD 184,000 - 357,000

Full time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

NVIDIA is seeking a Senior Software Engineer to lead bring-up, triage, benchmarking, and optimization of distributed AI workloads across GPU platforms at scale. You will set technical direction across communication libraries, model frameworks, and inference/training stacks, driving reliability and performance for state-of-the-art LLM workloads.

You will own deep performance investigations, define benchmarking methodologies, and build resilience tools for multi-GPU, multi-node deployments.

Qualifications

  • Experience debugging and triaging AI applications across full stack—from app to hardware.
  • Expertise with NCCL, CUDA-aware distributed execution, multi-GPU/multi-node workloads at scale.
  • Strong Python and C/C++ programming skills; experience with containerized clusters.

Responsibilities

  • Lead bring-up, validation, and debugging of large-scale AI clusters, infrastructure, and workloads.
  • Benchmark and tune pre-training, post-training, and inference workloads using NVIDIA AI stacks.
  • Profile and optimize end-to-end workload performance across compute, memory, networking, and communication layers.
  • Analyze scaling efficiency for distributed LLM workloads and translate findings into tuning guidance.
  • Own root-cause analysis of hangs, regressions, and topology sensitivity in large distributed environments.
  • Define and build resilience and failure-attribution stacks for cluster-wide diagnosis.
  • Build repeatable benchmark suites, automation, and qualification workflows on new platforms.
  • Tune runtime settings and deployment configurations with framework and platform teams.
  • Deliver data-driven recommendations based on profiling and benchmarks.
  • Mentor engineers and drive technical standards across the organization.

Skills

Python
C/C++
Distributed systems
NCCL
CUDA
Performance profiling
Debugging
Leadership

Education

Bachelor's or Master's in Computer Science or related field

Tools

Nsight Systems
NCCL tests
Megatron/TensorRT tooling
TensorRT-LLM

Job description

NVIDIA is seeking a Senior Software Engineer to lead bring-up, triage, benchmarking, and optimization of distributed AI workloads across GPU platforms at scale. You will set technical direction across communication libraries, model frameworks, and inference/training stacks, driving reliability and performance for state-of-the-art LLM workloads.

You will own deep performance investigations, define benchmarking methodologies, and build resilience tools for multi-GPU, multi-node deployments.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior AI Systems Engineer: Distributed Training
Senior AI Systems Engineer: Distributed Training

NVIDIA Corporation • Santa Clara (CA)

On-site
USD 184,000 - 357,000
Equity
Benefits
Senior AI Runtime Systems Engineer - GPU Training
Senior AI Runtime Systems Engineer - GPU Training

The Briefed • Mountain View (CA)

On-site
USD 190,000 - 240,000
Equity grants
Bonus eligibility
Healthcare coverage
+2
Senior AI Systems Performance Engineer
Senior AI Systems Performance Engineer

NVIDIA Corporation • Austin (TX)

On-site
USD 272,000 - 431,250
Equity
Benefits
Senior AI Infra Engineer — Distributed Training & Inference
Senior AI Infra Engineer — Distributed Training & Inference

NVIDIA Corporation • Santa Clara (CA)

On-site
USD 184,000 - 356,500
Equity
Comprehensive benefits package
Senior AI GPU Performance & Optimization Engineer
Senior AI GPU Performance & Optimization Engineer

NVIDIA Corporation • United States

Hybrid
USD 150,000 - 351,000
Equity
Benefits
Senior HPC & AI Performance Engineer
Senior HPC & AI Performance Engineer

NVIDIA Corporation • United States

Remote
USD 140,000 - 190,000
Senior AI Systems Performance Engineer (Equity Eligible)
Senior AI Systems Performance Engineer (Equity Eligible)

NVIDIA Corporation • United States

Remote
USD 224,000 - 431,000
Equity
Benefits
Senior Systems Software Engineer: AI Infra & Kubernetes
Senior Systems Software Engineer: AI Infra & Kubernetes

NVIDIA Corporation • Seattle (WA)

Hybrid
USD 184,000 - 357,000
Equity
Health benefits
Flexible work arrangement
+1
Senior Performance Engineer: AI Workload Optimization
Senior Performance Engineer: AI Workload Optimization

NVIDIA Corporation • Redmond (WA)

On-site
USD 224,000 - 431,250
Equity
Benefits
Senior AI Training Performance Architect — Optimize at Scale
Senior AI Training Performance Architect — Optimize at Scale

NVIDIA Corporation • Santa Clara (CA)

On-site
USD 210,000 - 340,000
Equity
Benefits