Distributed AI Systems Engineer - LLM Benchmarking

NVIDIA Corporation; Santa Clara (CA); On-site; USD 108,000 - 196,000; Full time

Benefits offered by this job

Equity

Job summary

NVIDIA is seeking a Software Engineer to bring up, triage, benchmark, and optimize distributed training and inference workloads across GPU platforms at scale.

You will work on multi-GPU and multi-node LLM workloads, build benchmarking tooling, and collaborate with framework, systems, and platform teams to deliver data-driven recommendations.

A strong background in Python and C/C++, CUDA-enabled distributed execution, and debugging at scale is required. Equity and benefits are included.

Qualifications

• Bachelor’s or Master’s in Computer Science or related field (or equivalent experience).
• Experience developing software for AI, HPC, or systems-level applications.
• Hands-on with multi-GPU or multi-node workloads and CUDA-aware distributed execution.
• Experience debugging and scaling distributed systems.
• Experience debugging AI applications across the full stack, from the application level toward the hardware.
• Experience operating workloads in scheduled, containerized cluster environments.
• Excellent analytical, debugging, and communication skills, and a collaborative approach across teams.
• Strong Python and C/C++ programming skills.

Responsibilities

• Bring up, validate, and debug large-scale AI clusters and end-to-end workloads.
• Tune and benchmark AI pre-training, post-training, and inference workloads.
• Perform root-cause analysis of failures in distributed environments.
• Develop resilience and fault-attribution tooling for node, fabric, and workload failures.
• Build and maintain repeatable benchmark suites and qualification workflows.
• Tune runtime settings and deployment configurations with partner teams.
• Deliver data-driven recommendations from profiling and benchmarks.

Skills

Python

C/C++

CUDA

Multi-GPU

Distributed systems

Debugging

Communication

Education

Bachelor’s or Master’s in Computer Science or related field

Tools

NCCL

UCX

InfiniBand

RoCE

libfabric

TensorRT

Job description

NVIDIA is seeking a Software Engineer to bring up, triage, benchmark, and optimize distributed training and inference workloads across GPU platforms at scale.

You will work on multi-GPU and multi-node LLM workloads, build benchmarking tooling, and collaborate with framework, systems, and platform teams to deliver data-driven recommendations.

A strong background in Python and C/C++, CUDA-enabled distributed execution, and debugging at scale is required. Equity and benefits are included.