Senior AI Systems Engineer: Distributed Training

NVIDIA Corporation

Santa Clara (CA)

On-site

USD 184,000 - 357,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Benefits offered by this job

Equity
Benefits

Job summary

NVIDIA Corporation in Santa Clara, CA is seeking a Senior Software Engineer to lead bring-up, benchmarking, and optimization of distributed training and inference workloads across GPU platforms at scale.

You will set technical direction across communication libraries and model frameworks, conduct root-cause analysis, build repeatable benchmarks, and mentor engineers while driving performance and reliability for multi-node AI clusters.

Qualifications

  • 8+ years of experience developing software infrastructure for large-scale AI or HPC systems.
  • Expertise debugging and triaging AI applications across the full stack—from application to hardware.
  • Expert-level Python and C/C++ programming skills; experience with containerized clusters.

Responsibilities

  • Lead bring-up, validation, and debugging of large-scale AI clusters, infrastructure, and end-to-end workloads.
  • Bring up, tune, and benchmark AI pre-training, post-training, and inference workloads using PyTorch, NeMo / Megatron, TensorRT-LLM, and adjacent NVIDIA AI software stacks.
  • Profile and optimize end-to-end workload performance across compute, memory, networking, and communication layers.
  • Own root‑cause analysis of complex failures — hangs, performance regressions, topology sensitivity in large distributed environments.
  • Define and build resilience and failure‑attribution stack: detecting, triaging, and attributing node, fabric, and workload failures.

Skills

Python
C/C++
Distributed systems
Debugging

Education

Bachelor’s or Master’s in CS or related field

Tools

NCCL
CUDA
NVIDIA AI software stacks

Job description

NVIDIA Corporation in Santa Clara, CA is seeking a Senior Software Engineer to lead bring-up, benchmarking, and optimization of distributed training and inference workloads across GPU platforms at scale.

You will set technical direction across communication libraries and model frameworks, conduct root-cause analysis, build repeatable benchmarks, and mentor engineers while driving performance and reliability for multi-node AI clusters.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior AI Infra Engineer — Distributed Training & Inference
Senior AI Infra Engineer — Distributed Training & Inference

NVIDIA Corporation • Santa Clara (CA)

On-site
USD 184,000 - 356,500
Equity
Comprehensive benefits package
Senior AI Training Performance Architect — Optimize at Scale
Senior AI Training Performance Architect — Optimize at Scale

NVIDIA Corporation • Santa Clara (CA)

On-site
USD 210,000 - 340,000
Equity
Benefits
Senior AI Runtime Systems Engineer - GPU Training
Senior AI Runtime Systems Engineer - GPU Training

The Briefed • Mountain View (CA)

On-site
USD 190,000 - 240,000
Equity grants
Bonus eligibility
Healthcare coverage
+2
Senior Cloud AI Infrastructure Engineer (Equity)
Senior Cloud AI Infrastructure Engineer (Equity)

NVIDIA Corporation • Town of Montana (WI)

On-site
USD 152,000 - 288,000
Equity
Benefits
Senior AI Networking & Performance Engineer
Senior AI Networking & Performance Engineer

NVIDIA Corporation • Colorado

On-site
USD 272,000 - 431,250
Senior HPC & AI Performance Engineer
Senior HPC & AI Performance Engineer

NVIDIA Corporation • United States

Remote
USD 140,000 - 190,000
Senior AI Infrastructure Engineer — GPU & Datacenter
Senior AI Infrastructure Engineer — GPU & Datacenter

NVIDIA Corporation • Santa Clara (CA)

On-site
USD 224,000 - 431,000
Equity
Benefits
Senior Systems Software Engineer: AI Infra & Kubernetes
Senior Systems Software Engineer: AI Infra & Kubernetes

NVIDIA Corporation • Seattle (WA)

Hybrid
USD 184,000 - 357,000
Equity
Health benefits
Flexible work arrangement
+1
Senior AI Inference Systems Engineer
Senior AI Inference Systems Engineer

NVIDIA Corporation • Santa Clara (CA)

Hybrid
USD 224,000 - 431,250
Equity
Benefits
Senior ML Training Systems Engineer - Distributed CUDA
Senior ML Training Systems Engineer - Distributed CUDA

Genesis AI • San Francisco (CA)

On-site
USD 180,000 - 260,000