GPU Infrastructure Engineer - Live-Cluster Support

Rune; Mountain View (CA); On-site; USD 175,000 - 260,000; Full time

Job summary

Rune, Mountain View, CA, seeks a hands-on engineer to own customer-facing technical support for our GPU clusters. You will diagnose issues, respond to customers, and ensure clusters perform as expected without handing off technical judgments.

You’ll tackle networking, node provisioning, GPU health, and firmware compatibility across the fleet, coordinate with support teams, and contribute to improving provisioning defaults and incident updates for client engineering teams.

Qualifications

• Hands-on experience with RDMA fabrics — RoCEv2, including ECN/DCQCN, PFC, and DCBX configuration.
• Experience debugging multi-node GPU interconnect performance with NCCL and RDMA benchmarking tools like perftest and nccl-tests.
• Working knowledge of GPU interconnect topology and how it affects distributed training performance.
• Experience reading GPU health telemetry (NVIDIA DCGM, nvidia-smi) and managing driver/firmware compatibility.
• Strong Linux systems administration and fleet automation experience (SSH security, Ansible).
• Comfortable owning live, customer-facing escalations on production systems.

Responsibilities

• Serve as the primary technical responder for customer-reported networking, GPU-interconnect, and node issues on live clusters.
• Diagnose multi-node NCCL and RDMA performance regressions and distinguish between hardware, driver, or configuration issues.
• Configure and validate RoCEv2 congestion control and reconcile host NIC settings with the switch fabric's behavior.
• Diagnose GPU hardware faults using DCGM telemetry and Xid error codes, distinguishing hardware failures from software issues.
• Own driver and firmware version compatibility across the fleet and validate before/after updates.
• Provision, re-provision, and decommission GPU nodes with proper testing and acceptance procedures.
• Triage edge-network and jumpbox access issues affecting provisioning workflows.
• Write incident updates and post-resolution summaries for customer engineering teams.

Skills

RDMA fabrics

NVIDIA ConnectX NICs

NCCL

NVIDIA DCGM

GPU health telemetry

Linux administration

Ansible

SSH security

Tools

perftest

nccl-tests

OpenMPI/Slurm integration

Ansible playbooks

Job description

Rune, Mountain View, CA, seeks a hands-on engineer to own customer-facing technical support for our GPU clusters. You will diagnose issues, respond to customers, and ensure clusters perform as expected without handing off technical judgments.

You’ll tackle networking, node provisioning, GPU health, and firmware compatibility across the fleet, coordinate with support teams, and contribute to improving provisioning defaults and incident updates for client engineering teams.