Senior Solutions Architect, AI Factory Observability and Visualization - NVIS

NVIDIA Corporation; Oregon; Hybrid; USD 184,000 - 357,000; Full time

Benefits offered by this job

Equity

Benefits & programs

Job summary

NVIDIA Corporation is seeking a Senior Solutions Architect for AI Factory Observability & Visualization. This remote role focuses on full-spectrum visibility across HPC and AI factories, transforming telemetry into actionable insights from hardware to workload.

You will validate systems, understand end-to-end architecture, and lead telemetry development to ensure healthy operation and rapid issue resolution.

Qualifications

• Bachelor's degree or equivalent in CS/Math/Engineering/Physics or related field.
• 6+ years of experience managing Linux-based HPC/distributed/AI infrastructure.
• Hands-on experience with multi-GPU and multi-node clusters, including networking/interconnects.
• Proficiency with Python and Shell for scripting, automation, and tooling.
• Experience with observability stacks (Prometheus, Grafana, Loki) including exporters/collectors.
• Experience transforming metrics, logs, traces into actionable insights for distributed environments.
• Strong cross-functional communication skills.

Responsibilities

• Run AI factory validation tools and microbenchmarks; interpret results for health and performance.
• Understand system from network topology to compute end-to-end.
• Define healthy state with metrics, logs, and signals and thresholding.
• Build/extend telemetry across hardware, fabric, and workloads; define data collection and storage.
• Serve as observability expert; identify gaps in visibility and reflect true behavior.
• Develop automation (Python, Shell) for collecting and presenting system data.
• Recommend improvements to visibility sources and reporting for clarity.
• Collaborate across hardware, software, networking, and datacenter teams for customer readiness.

Skills

Observability architecture

Python scripting

Shell scripting

HPC systems

Distributed systems

Networking & interconnects

Multi-GPU clusters

Telemetry & data pipelines

Education

Bachelor's degree or equivalent in CS/Math/Engineering/Physics

Tools

Prometheus

Grafana

Loki

DCGM

NVLink

InfiniBand

Job description

NVIDIA's Infrastructure Specialists team is hiring a Senior Solutions Architect - AI Factory Observability & Visualization! This remote role develops full-spectrum visibility that supports the smooth functioning of HPC systems and AI factories, transforming intricate telemetry across network and compute into straightforward, actionable perspectives.The role has a complete, end-to-end understanding of the HPC/AI system, running and interpreting microbenchmarks and workloads to confirm system readiness, then establishing the observability that maintains this state. The work involves collaborating across NVIDIA teams to help partners see, understand, and respond to HPC system and AI factory performance, from hardware to workload.

What You Will be Doing:

• Run AI factory validation tools, microbenchmarks, and workloads provided by the team, and interpret results to assess system health and performance.
• Gain a comprehensive understanding of the system from start to finish, including network topology, interconnects, and compute.
• Establish what "healthy" represents across the stack — the metrics, logs, and signals that confirm a system is functioning well, and the thresholds that show it isn't.
• Build and extend the telemetry surface across hardware, fabric, and workload, crafting how data is collected, transformed, stored, and surfaced.
• Serve as the observability expert, investigating gaps in visibility to ensure it reflects true system behavior.
• Develop automation (Python, Shell) for collecting, transforming, and presenting system and network data.
• Recommend improvements to system visibility, data sources, and reporting that give teams clearer insight.
• Collaborate with hardware, software, networking, datacenter, and product groups to ready HPC systems and AI factories for customer deployment, contributing documentation and readiness materials throughout the process.

What We Need to See:

• Bachelor's degree or equivalent experience in Computer Science, Mathematics, Engineering, Physics, or related field.
• 6+ years of experience managing Linux-based systems in HPC, distributed systems, or large AI/ML settings.
• Hands-on experience with the architecture of multi-GPU and/or multi-node clusters, including networking and interconnects.
• Solid grasp of how HPC and AI factory systems fit together end to end, from network fabric through compute.
• Proficiency with Python and Shell/Bash for scripting, automation, and tooling.
• Practical experience working with observability systems (e.g., Prometheus, Grafana, Loki, or similar), including building custom exporters or collectors, setting up alerts, and handling metric cardinality and retention on a large scale.
• Experience transforming metrics, logs, and traces into clear, actionable insight for complex distributed environments.
• Familiarity with GPU and fabric telemetry (e.g., DCGM, NVLink, InfiniBand/Ethernet fabric counters) and using it to diagnose performance regressions.
• Strong communication skills and the ability to work effectively with cross-functional teams.

Ways to Stand Out From the Crowd:

• Experience with AI factory or large-scale AI infrastructure build, deployment, or operations.
• Background in HPC systems engineering, SRE, or systems analysis for GPU-accelerated environments.
• Experience building automation and data pipelines that feed dashboards and reporting at scale.
• Demonstrated desire to use AI to solve practical problems, improve workflows, and guide data-driven decisions.

NVIDIA is widely considered one of the technology world's most desirable employers. Some of the world's most forward-thinking and hardworking people are working for us.

Your base salary will be determined based on your location, experience, and other objective job-related factors.

The expected base salary ranges for this position are:184,000 USD - 287,500 USD for Level 4; and 224,000 USD - 356,500 USD for Level 5.

You will also be eligible for equity and benefits. Applications for this job will be accepted at least until November 3, 2026.

This posting is for an existing vacancy.

NVIDIA uses AI tools in its recruiting processes.

NVIDIA is committed to fostering an inclusive work environment and proud to be an equal opportunity employer. As we highly value diversity in our current and future employees, we do not discriminate (including in our hiring and promotion practices) on the basis of race, religion, color, national origin, gender, gender expression, sexual orientation, age, marital status, veteran status, disability status or any other characteristic protected by law.