Senior AI Factory Architect - HPC and NCCL Benchmarks

NVIDIA; Durham (NC); On-site; USD 152,000 - 287,500; Full time

Benefits offered by this job

Equity

Benefits

Job summary

NVIDIA is seeking a Senior Solutions Architect to join the Infrastructure Specialists team in Durham, NC. You will design, implement, and validate AI factories, optimizing AI/LLM workloads on Linux GPU clusters and contributing to observability and automation efforts.

You will mentor under senior architects, collaborate across teams, and help prepare customer-ready AI factories by validating hardware and software for current AI applications.

Qualifications

• Bachelor’s degree or equivalent in a relevant field.
• 5+ years of experience managing Linux-based systems in HPC, distributed systems, or AI/ML environments.
• Hands-on experience running AI/ML workloads on multi-GPU and multi-node clusters, with exposure to NCCL.
• Practical knowledge of collective communication patterns like AllReduce and AllToAll in ML/LLM training.
• Proficiency in Python and Shell/Bash for scripting, automation, and tooling.
• Strong communication skills and ability to work with cross-functional teams.

Responsibilities

• Set up, verify, and optimize AI factory environments on multi-GPU, multi-node Linux clusters.
• Validate configurations for NCCL, collectives, and distributed training frameworks.
• Run AI/LLM benchmarks: setup, orchestration, result collection, analysis.
• Troubleshoot training jobs or benchmarks that fail or underperform.
• Build observability: metrics, logs, traces, dashboards for workload understanding.
• Automate benchmarks and regression checks using Python and Shell.
• Analyze NCCL usage and collectives to improve throughput and latency.
• Suggest improvements to job configuration, parallelism, and cluster settings.
• Collaborate with hardware, software, networking, and product teams to prepare AI factories for customers.
• Contribute to internal and customer-facing readiness materials.

Skills

Python

Shell/Bash

Communication

Education

Bachelor's degree or equivalent (CS/Engineering/Math/Physics)

Tools

NCCL

Job description

NVIDIA is seeking a Senior Solutions Architect to join the Infrastructure Specialists team in Durham, NC. You will design, implement, and validate AI factories, optimizing AI/LLM workloads on Linux GPU clusters and contributing to observability and automation efforts.

You will mentor under senior architects, collaborate across teams, and help prepare customer-ready AI factories by validating hardware and software for current AI applications.