Remote Senior AI Factory Observability Architect

NVIDIA Corporation

United States

Remote

USD 184,000 - 357,000

Full time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

NVIDIA Corporation is seeking a Senior Solutions Architect to drive end-to-end observability for AI factory and HPC systems. You will define healthy state, build telemetry across hardware, fabric, and workloads, and collaborate across teams to prepare deployments for customers.

Strong Python/Shell scripting and experience with Prometheus, Grafana, and GPU telemetry are essential. The role supports a remote work model with global impact, offering equity and benefits.

Qualifications

  • Bachelor’s degree or equivalent experience in Computer Science, Mathematics, Engineering, Physics, or related field.
  • 6+ years of experience managing Linux-based systems in HPC, distributed systems, or large AI/ML settings.
  • Hands-on experience with the architecture of multi-GPU and/or multi-node clusters, including networking and interconnects.
  • Solid grasp of how HPC and AI factory systems fit together end to end, from network fabric through compute.
  • Proficiency with Python and Shell/Bash for scripting, automation, and tooling.
  • Practical experience working with observability systems (e.g., Prometheus, Grafana, Loki, or similar), including building custom exporters or collectors, setting up alerts, and handling metric cardinality and retention on a large scale.
  • Experience transforming metrics, logs, and traces into clear, actionable insight for complex distributed environments.
  • Familiarity with GPU and fabric telemetry (e.g., DCGM, NVLink, InfiniBand/Ethernet fabric counters) and using it to diagnose performance regressions.
  • Strong communication skills and the ability to work effectively with cross-functional teams.

Responsibilities

  • Run AI factory validation tools, microbenchmarks, and workloads provided by the team, and interpret results to assess system health and performance.
  • Gain a comprehensive understanding of the system from start to finish, including network topology, interconnects, and compute.
  • Establish what “healthy” represents across the stack — the metrics, logs, and signals that confirm a system is functioning well, and the thresholds that show it isn’t.
  • Build and extend the telemetry surface across hardware, fabric, and workload, crafting how data is collected, transformed, stored, and surfaced.
  • Serve as the observability expert, investigating gaps in visibility to ensure it reflects true system behavior.
  • Develop automation (Python, Shell) for collecting, transforming, and presenting system and network data.
  • Recommend improvements to system visibility, data sources, and reporting that give teams clearer insight.
  • Collaborate with hardware, software, networking, datacenter, and product groups to ready HPC systems and AI factories for customer deployment, contributing documentation and readiness materials throughout the process.

Skills

Python
Shell
Linux
HPC
Distributed systems
Networking

Education

Bachelor’s degree or equivalent experience

Tools

Prometheus
Grafana
Loki

Job description

NVIDIA Corporation is seeking a Senior Solutions Architect to drive end-to-end observability for AI factory and HPC systems. You will define healthy state, build telemetry across hardware, fabric, and workloads, and collaborate across teams to prepare deployments for customers.

Strong Python/Shell scripting and experience with Prometheus, Grafana, and GPU telemetry are essential. The role supports a remote work model with global impact, offering equity and benefits.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Remote Senior AI Factory Observability Architect
Remote Senior AI Factory Observability Architect

NVIDIA Corporation • Santa Clara (CA)

Remote
USD 184,000 - 356,500
Equity options
Comprehensive benefits package
Remote Senior AI Factory Observability Architect
Remote Senior AI Factory Observability Architect

NVIDIA Corporation • Durham (NC)

Remote
USD 184,000 - 287,500
Equity
Benefits
Remote Senior AI Factory Observability Architect
Remote Senior AI Factory Observability Architect

NVIDIA Corporation • Austin (TX)

Remote
USD 184,000 - 287,500
Equity
Benefits
Remote Senior AI Factory Observability Architect
Remote Senior AI Factory Observability Architect

NVIDIA Corporation • California

Remote
USD 184,000 - 287,500
Equity options
Health benefits
Senior Solutions Architect, AI Factory Observability and Visualization - NVIS
Senior Solutions Architect, AI Factory Observability and Visualization - NVIS

NVIDIA Corporation • California

Remote
USD 184,000 - 287,500
Equity options
Health benefits
Senior AI/HPC Observability Engineer
Senior AI/HPC Observability Engineer

NVIDIA Corporation • Santa Clara (CA)

On-site
USD 184,000 - 288,000
Equity
Benefits
Senior Solutions Architect, AI Factory Observability and Visualization - NVIS
Senior Solutions Architect, AI Factory Observability and Visualization - NVIS

NVIDIA Corporation • United States

Remote
USD 184,000 - 357,000
Senior AI and HPC Observability Engineer
Senior AI and HPC Observability Engineer

NVIDIA Corporation • Santa Clara (CA)

On-site
USD 184,000 - 288,000
Equity
Benefits
Senior AI Infrastructure & Observability Engineer
Senior AI Infrastructure & Observability Engineer

NVIDIA Corporation • Westford (MA)

On-site
USD 184,000 - 357,000
Equity
Benefits
Senior AIOps & Observability Engineer — Equity Eligible
Senior AIOps & Observability Engineer — Equity Eligible

NVIDIA Corporation • Santa Clara (CA)

On-site
USD 200,000 - 322,000
Equity
Benefits