DGX Cloud AI Infra Engineer - Benchmark & Debug

NVIDIA; Santa Clara (CA); On-site; USD 150,000 - 190,000; Full time

Benefits offered by this job

Equity

Benefits

Job summary

NVIDIA is seeking a Software Engineer to bring up, benchmark, and optimize distributed training and inference workloads across its GPU platforms at large scale. This hands-on role focuses on debugging, benchmarking, and improving multi-node AI workloads using CUDA-enabled stacks.

The ideal candidate will have strong Python/C++ skills, experience with multi-GPU and multi-node deployments, and a track record of debugging distributed systems. Equity and benefits are provided.

Qualifications

• Bachelor’s or Master’s in Computer Science or a related technical field (or equivalent experience).
• Experience developing software for AI, HPC, or systems-level applications.
• Hands-on experience with multi-GPU or multi-node workloads and CUDA-aware distributed execution.
• Background with debugging and scaling distributed systems.
• Experience debugging and triaging AI applications across the full stack, from the application level toward the hardware.
• Experience operating workloads in scheduled, containerized cluster environments.
• Excellent analytical, debugging, and communication skills, and a collaborative approach across teams.
• Strong Python and C/C++ programming skills.

Responsibilities

• Bring up, validate, and debug large-scale AI clusters, infrastructure, and end-to-end workloads.
• Bring up, tune, and benchmark AI pre-training, post-training, and inference workloads using PyTorch, NeMo / Megatron, TensorRT-LLM, and adjacent NVIDIA AI software stacks.
• Perform root-cause analysis of failures in large distributed environments
• Contribute to the resilience and failure-attribution tooling that detects, triages, and attributes node, fabric, and workload failures across the cluster.
• Build and maintain repeatable benchmark suites, automation, acceptance criteria, and qualification workflows on new platforms.
• Tune runtime settings, communication parameters, and deployment configurations in close partnership with framework, systems, and platform teams.
• Deliver actionable, data-driven recommendations based on profiling, benchmark results, and cluster characterization.

Skills

Python

C/C++

Distributed systems

GPU performance

Data-driven analysis

Education

Bachelor’s or Master’s in Computer Science or related

Tools

NCCL

CUDA-aware distributed execution

UCX

libfabric

InfiniBand / RoCE

Job description

NVIDIA is seeking a Software Engineer to bring up, benchmark, and optimize distributed training and inference workloads across its GPU platforms at large scale. This hands-on role focuses on debugging, benchmarking, and improving multi-node AI workloads using CUDA-enabled stacks.

The ideal candidate will have strong Python/C++ skills, experience with multi-GPU and multi-node deployments, and a track record of debugging distributed systems. Equity and benefits are provided.