DSX NVCRE Senior Engineer

NVIDIA; United States; Remote; USD 184,000 - 357,000; Full time

Benefits offered by this job

Equity

Benefits

Job summary

NVIDIA Corporation seeks a Senior Software Engineer to join the DSX team to build tooling for validating AI infrastructure ready for production. You will design cluster readiness software, implement controllers and CLIs for Kubernetes and Slurm, and develop benchmarks and provenance pipelines to ensure verifiable results.

Responsibilities include integration across GPUs, schedulers, and fabrics, cross-functional collaboration, and contributing to open-source projects.

Qualifications

• Degree in computer science/engineering or related field.
• 8+ years of software engineering experience building infrastructure or systems software.
• Strong Go and Python programming with production-grade software experience.
• Deep familiarity with Linux, containers, and Kubernetes; experience writing operators/controllers.
• Experience with HPC schedulers like Slurm and GPU cluster fundamentals.
• Experience benchmarking/validating distributed systems with rigorous measurement and reproducibility.
• Ability to drive investigations to root cause and communicate findings clearly.

Responsibilities

• Design and build cluster readiness and certification tooling for production validation.
• Develop Go and Python tooling running on Kubernetes and Slurm (controllers, operators, CLIs, integrations).
• Build benchmarks and workloads covering NCCL, compute, storage, diagnostics, training, inference, reliability, and security.
• Create provenance pipelines to capture run details and generate auditable results.
• Integrate new GPUs, schedulers, and fabrics from bring-up to multi-node validation.
• Collaborate with cross-functional teams and cloud partners to address gaps and improve platforms.
• Contribute to open-source Cluster Readiness Engine and related projects.

Skills

Go

Python

Linux

Kubernetes

Slurm

Distributed systems

Root-cause analysis

Education

BS/MS/PhD in CS/EE or related field

Tools

DCGM

Job description

NVIDIA is seeking a Senior Software Engineer to join our DSX team to build and operate the tooling that verifies AI infrastructure is ready for production. You will work on cluster readiness and certification software, including the open-source Cluster Readiness Engine, that validates GPU clusters across Kubernetes and Slurm, measures real training and inference performance, and produces evidence customers and partners can trust. We partner with the world's most innovative AI companies and cloud providers to make sure their infrastructure performs as designed before their most important workloads land on it.

What you will be doing:

• Shape the Future of Validation. Design and build the systems that turn cluster validation from a manual exercise into repeatable, verifiable software. Your work will directly shape how NVIDIA and its partners qualify new GPU platforms and deployments.
• Core Development. Design and implement cluster readiness and certification tooling in Go and Python that runs on Kubernetes and Slurm, including controllers, operators, CLIs, and scheduler integrations.
• Benchmarking & Workloads. Build a catalog of benchmarks and real workloads spanning communication (NCCL), compute, storage, diagnostics, training, inference, reliability, and security, with per-architecture expectations and clear pass/fail criteria.
• Data Provenance. Develop provenance and evidence pipelines that capture what actually ran, where it ran, and under which configuration, and that produce signed, auditable results.
• Platform Integration. Bring new GPU platforms, schedulers, and fabrics under test, from first bring-up on a single node through multi-node validation on production clusters.
• Cross-functional Collaboration.Collaborate with internal platform, networking, and solutions teams and with cloud partners to root-cause performance and correctness gaps found by the tooling, and feed fixes back into platforms and reference architectures.
• Open Source Contribution. Contribute to the open-source Cluster Readiness Engine and related projects, including design documents, documentation, and reviews.
• Technical Authorship. Author design documents, runbooks, and operational guides that codify how clusters are validated and how results are interpreted.

What we need to see:

• BS, MS, or Ph.D. in Computer Science, Computer/Electrical Engineering, or a related technical field, or equivalent experience.
• 8+ years of software engineering experience building infrastructure or systems software, ideally for large-scale compute, cloud, and HPC environments.
• Strong programming skills in Go and Python, with experience shipping production-quality services, controllers, or command-line tools.
• Deep familiarity with Linux systems, containers, and Kubernetes, including writing operators or controllers against the Kubernetes API.
• Working knowledge of HPC schedulers such as Slurm and of GPU cluster fundamentals: drivers, NCCL, InfiniBand or RoCE fabrics, and GPU health tooling such as DCGM.
• Experience benchmarking or validating distributed systems, with a rigorous approach to measurement, reproducibility, and evidence.
• Demonstrated ability to drive complex technical investigations to root cause and to communicate findings clearly to engineering and leadership audiences.

Ways to stand out from the crowd:

• Hands-on experience running or tuning large-scale distributed training or inference workloads with frameworks such as PyTorch, Megatron, NeMo, vLLM, SGLang, or TensorRT-LLM.
• Experience with the NVIDIA ecosystem, including DGX systems, GB200/GB300 platforms, CUDA, NVLink and NVSwitch fabrics, and NVIDIA networking.
• Prior contributions to open-source infrastructure projects, especially Kubernetes operators, scheduler plugins, or benchmarking frameworks.
• Background in software supply-chain security, attestation, or signed artifacts, and in building audit-grade evidence into systems.
• Familiarity with observability stacks (Prometheus, Grafana, OpenTelemetry), GitOps workflows, and infrastructure as code for GPU clusters.

NVIDIA is leading the way in groundbreaking developments in Artificial Intelligence, High-Performance Computing, and Visualization. The GPU, our invention, serves as the visual cortex of modern computers and is at the heart of our products and services. Our work opens up new universes to explore, enables amazing creativity and discovery, and powers what were once science fiction inventions from artificial intelligence to autonomous cars. NVIDIA is looking for phenomenal people like you to help us accelerate the next wave of artificial intelligence. NVIDIA is widely considered to be one of the technology world's most desirable employers. We have some of the most forward-thinking and dedicated people in the world working for us.

Your base salary will be determined based on your location, experience, and the pay of employees in similar positions. The base salary range is 184,000 USD - 287,500 USD for Level 4, and 224,000 USD - 356,500 USD for Level 5.

You will also be eligible for equity and benefits.

Applications for this job will be accepted at least until November 1, 2026.

This posting is for an existing vacancy.

NVIDIA uses AI tools in its recruiting processes.

NVIDIA is committed to fostering an inclusive work environment and proud to be an equal opportunity employer. As we highly value diversity in our current and future employees, we do not discriminate (including in our hiring and promotion practices) on the basis of race, religion, color, national origin, gender, gender expression, sexual orientation, age, marital status, veteran status, disability status or any other characteristic protected by law.