Senior System Architect, Infrastructure Reliability

NVIDIA Gruppe; Santa Clara (CA); On-site; USD 184,000 - 287,500; Full time

Benefits offered by this job

Equity

Comprehensive benefits

Job summary

NVIDIA Gruppe is seeking a Senior System Architect to develop a failure attribution framework for EDA systems. You will work on correlating hardware and software failures in real-time across heterogeneous nodes.

Ideal candidates will possess strong C++ and Python skills, along with deep knowledge of distributed systems and CPU architecture. The compensation ranges from 184,000 USD to 356,500 USD based on experience, and the role offers equity and benefits.

Qualifications

• 6+ years in systems programming.
• Experience in building automated RCA pipelines for HPC or cloud-scale.
• Strong knowledge of x86/ARM node-level metrics.

Responsibilities

• Develop an automated framework for telemetry ingestion.
• Implement diagnostics and correlation for GPU errors.
• Work on resiliency engineering with hardware teams.

Skills

Distributed Systems Mastery

Programming Proficiency in C++ and Python

CPU Architecture Knowledge

Education

BS, MS, or PhD in Computer Science or Electrical Engineering

Tools

NVIDIA DCGM

NVIDIA Management Library (NVML)

Slurm

Kubernetes

Job description

Senior System Architect: Heterogeneous EDA Systems

NVIDIA is seeking an engineer to solve a complex challenge in accelerated computing: Failure Attribution at Scale. As EDA or equivalent workloads scale across thousands of heterogeneous nodes, a single failure can cause massive resource waste. The role is to develop and build an automated framework that ingests telemetry from CPU and GPU clusters to identify the root cause of job failures in real‑time, distinguishing between hardware faults, infrastructure instability, and software defects.

What you’ll be doing

• Architect Failure Attribution Frameworks: Build a scalable 'flight recorder' for EDA jobs that captures high‑fidelity state across the CPU, GPU, and Fabric at the moment of failure.
• Build automated diagnostics that correlate GPU XID errors, PCIe bus failures, and CUDA memory exceptions. Connect these errors with system‑level events such as OOM kills or NUMA‑related hangs.
• Distributed Logging & Tracing: Implement low‑overhead tracing mechanisms (using tracing tools or custom agents) that provide access to job execution across multi‑node Slurm or Kubernetes clusters.
• Root Cause Automation: Develop heuristics and models based on machine learning to classify failures as 'Hardware Fault,' 'Software Bug,' or 'Environment Issue.' This reduces the Mean Time to Identify (MTTI) for R&D teams.
• Resiliency Engineering: Work closely with hardware and infrastructure teams to define 'signals of impending failure,' enabling proactive job migration or checkpointing before a crash occurs.

What we need to see

• Distributed Systems Mastery: BS, MS, or PhD in Computer Science or Electrical Engineering (or equivalent experience) with 6+ years in systems programming.
• Experience building automated RCA pipelines for HPC or cloud‑scale environments.
• CPU Architecture Deep‑Dive: Expert knowledge of x86/ARM node‑level metrics: IPC, cache contention, NUMA imbalance, and hardware interrupts.
• Programming Proficiency: Strong C++ and Python skills, with the ability to build high‑performance daemons that monitor system health without impacting workload performance.
• Scale Experience: Familiarity with cluster resource managers (Slurm, LSF, or Kubernetes) and how they manage job lifecycle and signal propagation.

Ways to Stand Out from the Crowd

• Low‑Level Diagnostics: Expert knowledge of the Linux kernel and its error‑reporting interfaces (/dev/mcelog, dmesg, journald). Understand how the kernel handles hardware exceptions and memory faults.
• GPU Infrastructure Proficiency: Deep experience with the NVIDIA DCGM and NVIDIA Management Library (NVML) for monitoring device health and capturing state‑dumps.
• Experience with tools doing non‑intrusive monitoring of application health and syscall‑level failure patterns.
• Experience with checkpoint/restore technologies (like CRIU) and their application in long‑running EDA flows.

Your base salary will be determined based on your location, experience, and the pay of employees in similar positions. The base salary range is 184,000 USD - 287,500 USD for Level 4, and 224,000 USD - 356,500 USD for Level 5.

You will also be eligible for equity and benefits.

NVIDIA uses AI tools in its recruiting processes.

NVIDIA is committed to fostering a diverse work environment and proud to be an equal‑opportunity employer. We do not discriminate on the basis of race, religion, color, national origin, gender, gender expression, sexual orientation, age, marital status, veteran status, disability status or any other characteristic protected by law.