Senior DL Infra Engineer: Scalable GPU AI Training

NVIDIA; California (MO); On-site; USD 224,000 - 431,000; Full time

Benefits offered by this job

Equity

Benefits

Job summary

NVIDIA seeks a Deep Learning Software Infrastructure Engineer to propel its Autonomous Vehicles project forward. You will build and scale training libraries and infrastructure, enabling training on thousands of GPUs and massive datasets to accelerate iteration and safety.

This role requires advanced degree and 12+ years in ML and distributed systems; expertise in PyTorch, NCCL, Slurm, Kubernetes, Lustre, and RoCE/IB networking.

Qualifications

• BS/MS/PhD in CS/EE or equivalent experience.
• 12+ years of professional experience in high‑performance distributed systems.
• Expertise in PyTorch; large‑scale training (DDP/FSDP, NCCL, tensor/pipeline parallelism).
• Strong systems background: datacenter networking (RoCE, IB); Lustre filesystems.
• Proficient in Python; production‑grade libraries, orchestration, and automation.
• Ability to collaborate with ML researchers and infra engineers.

Responsibilities

• Craft and scale deep learning infrastructure libraries for multi‑thousand GPU clusters.
• Improve training stack efficiency: data loaders, distributed training, scheduling, monitoring.
• Build robust training pipelines for massive video datasets and rapid experimentation.
• Collaborate with researchers and platform teams to enhance efficiency and availability.
• Own core components like orchestration libraries and fault‑tolerant training systems.
• Partner with leadership to scale infrastructure with growing GPU capacity while staying stable.

Skills

Distributed systems

Python

PyTorch

Large-scale training

NCCL

RoCE/IB networking

Lustre

Slurm

Kubernetes

Tensor/pipeline parallelism

Education

BS / MS / PhD in CS/EE or equivalent

Tools

Slurm

Kubernetes

Lustre

RoCE/IB

Job description

NVIDIA seeks a Deep Learning Software Infrastructure Engineer to propel its Autonomous Vehicles project forward. You will build and scale training libraries and infrastructure, enabling training on thousands of GPUs and massive datasets to accelerate iteration and safety.

This role requires advanced degree and 12+ years in ML and distributed systems; expertise in PyTorch, NCCL, Slurm, Kubernetes, Lustre, and RoCE/IB networking.