Senior AI Infra Engineer - Resilient Training & Open Source

NVIDIA Corporation

Santa Clara (CA)

On-site

USD 224,000 - 431,000

Full time

13 days ago

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

NVIDIA Corporation is seeking a Senior Software Engineer for the Recovery and Resilience team in Santa Clara, CA. You will own AI infrastructure services, tools, and open-source projects to support resilient and efficient training from design through production.

You will design, build and operate services and APIs, mentor engineers, and drive open-source initiatives across cloud and on-prem environments. Strong experience with Kubernetes/Slurm and PyTorch/TensorFlow is required.

Qualifications

  • BS/MS/PhD in Computer Science or related field plus 12+ years of experience
  • Experience taking large systems from design through production
  • Strong experience with Kubernetes and/or Slurm at scale
  • Hands-on experience with PyTorch, TensorFlow, JAX or Ray
  • Strong understanding of NVIDIA GPUs and networking, including how they fail
  • Experience running production services, including CI/CD, monitoring, alerting and on-call

Responsibilities

  • Design, build, launch and operate services, tools and open-source projects
  • Plan and mentor engineers, review designs
  • Build resilient AI infrastructure on Kubernetes and Slurm for training workloads
  • Support on-premises clusters and cloud supercomputers
  • Create APIs linking to resiliency systems and automate failures
  • Develop job submission layer to hide infrastructure failures and restart jobs
  • Build agentic tools to diagnose bottlenecks and suggest fixes
  • Lead open-source contributions and release components for community use
  • Define service governance: monitoring, alerts, CI/CD, gates, on-call plans

Skills

Kubernetes
Slurm at scale
PyTorch
TensorFlow
JAX
Ray
GPU networking

Education

BS/MS/PhD in Computer Science or related field

Tools

Docker
CI/CD

Job description

NVIDIA Corporation is seeking a Senior Software Engineer for the Recovery and Resilience team in Santa Clara, CA. You will own AI infrastructure services, tools, and open-source projects to support resilient and efficient training from design through production.

You will design, build and operate services and APIs, mentor engineers, and drive open-source initiatives across cloud and on-prem environments. Strong experience with Kubernetes/Slurm and PyTorch/TensorFlow is required.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior AI Infrastructure Engineer — Resilient GPU Training
Senior AI Infrastructure Engineer — Resilient GPU Training

Nvidia Corporation in • Santa Clara (CA)

On-site
USD 224,000 - 357,000
Equity
Benefits
Senior System Software Engineer, DGX Cloud AI Infrastructure
Senior System Software Engineer, DGX Cloud AI Infrastructure

Nvidia Corporation in • Santa Clara (CA)

On-site
USD 224,000 - 357,000
Equity
Benefits
Senior Cloud AI Infrastructure Engineer (Equity)
Senior Cloud AI Infrastructure Engineer (Equity)

NVIDIA Corporation • Town of Montana (WI)

On-site
USD 152,000 - 288,000
Equity
Benefits
Senior System Software Engineer, DGX Cloud AI Infrastructure
Senior System Software Engineer, DGX Cloud AI Infrastructure

2100 NVIDIA USA • Santa Clara (CA)

On-site
USD 224,000 - 431,000
Senior Software Engineer, AI Infra & Kubernetes
Senior Software Engineer, AI Infra & Kubernetes

Thomas To • Santa Clara (CA)

On-site
USD 184,000 - 357,000
Senior AI Systems Engineer: Distributed Training
Senior AI Systems Engineer: Distributed Training

NVIDIA Corporation • Santa Clara (CA)

On-site
USD 184,000 - 357,000
Equity
Benefits
Senior AI Infrastructure Engineer — GPU & Datacenter
Senior AI Infrastructure Engineer — GPU & Datacenter

NVIDIA Corporation • Santa Clara (CA)

On-site
USD 224,000 - 431,000
Equity
Benefits
Resiliency Architect: AI Hardware Diagnostics & Modeling
Resiliency Architect: AI Hardware Diagnostics & Modeling

NVIDIA Corporation • United States

Hybrid
USD 184,000 - 288,000
Equity
Benefits
Senior AI Training Infrastructure Engineer
Senior AI Training Infrastructure Engineer

NVIDIA Corporation • California

On-site
USD 224,000 - 431,000
Senior AI Data Infrastructure Engineer
Senior AI Data Infrastructure Engineer

NVIDIA Corporation • Santa Clara (CA)

On-site
USD 152,000 - 288,000
Equity
Comprehensive benefits
Career growth opportunities