GPU Infrastructure Engineer

Rune; Mountain View (CA); Sur place; USD 175 000 - 260 000; Plein temps

Résumé du poste

Rune, Mountain View, CA, seeks a hands-on engineer to own customer-facing technical support for our GPU clusters. You will diagnose issues, respond to customers, and ensure clusters perform as expected without handing off technical judgments.

You’ll tackle networking, node provisioning, GPU health, and firmware compatibility across the fleet, coordinate with support teams, and contribute to improving provisioning defaults and incident updates for client engineering teams.

Qualifications

• Hands-on experience with RDMA fabrics — RoCEv2, including ECN/DCQCN, PFC, and DCBX configuration.
• Experience debugging multi-node GPU interconnect performance with NCCL and RDMA benchmarking tools like perftest and nccl-tests.
• Working knowledge of GPU interconnect topology and how it affects distributed training performance.
• Experience reading GPU health telemetry (NVIDIA DCGM, nvidia-smi) and managing driver/firmware compatibility.
• Strong Linux systems administration and fleet automation experience (SSH security, Ansible).
• Comfortable owning live, customer-facing escalations on production systems.

Responsabilités

• Serve as the primary technical responder for customer-reported networking, GPU-interconnect, and node issues on live clusters.
• Diagnose multi-node NCCL and RDMA performance regressions and distinguish between hardware, driver, or configuration issues.
• Configure and validate RoCEv2 congestion control and reconcile host NIC settings with the switch fabric's behavior.
• Diagnose GPU hardware faults using DCGM telemetry and Xid error codes, distinguishing hardware failures from software issues.
• Own driver and firmware version compatibility across the fleet and validate before/after updates.
• Provision, re-provision, and decommission GPU nodes with proper testing and acceptance procedures.
• Triage edge-network and jumpbox access issues affecting provisioning workflows.
• Write incident updates and post-resolution summaries for customer engineering teams.

Connaissances

RDMA fabrics

NVIDIA ConnectX NICs

NCCL

NVIDIA DCGM

GPU health telemetry

Linux administration

Ansible

SSH security

Outils

perftest

nccl-tests

OpenMPI/Slurm integration

Ansible playbooks

Description du poste

Location: Mountain View, CA (not open to remote)

About Rune:

Every solar and wind power plant is a latent data center. Rune connects wasted power to the compute that needs it. Solar plants generate electricity the grid can't take: clipped by inverters, curtailed by operators, gone before it ever reaches a meter. Meanwhile, AI needs more power than new construction can deliver in time. Rune deploys hardware, software, and financial solutions to put this power to work. We build modular micro data centers that install directly at solar sites, bypassing the traditional grid entirely and turning stranded power into compute. We're backed by Spark Capital, Union Square Ventures, and Lowercarbon Capital.

The Role:

We're looking for a hands-on engineer to own customer-facing technical support for our GPU clusters: networking, node provisioning, and GPU health. You'll be the primary responder to ensure our customers' clusters are performing as expected, including diagnosing issues, resolving them, or routing them correctly, without needing to hand off the technical judgment to someone else.

Key Responsibilities

• Serve as the primary technical responder for customer-reported networking, GPU-interconnect, and node issues on live clusters.


• Diagnose multi-node NCCL and RDMA performance regressions, including issues sourced to GPUDirect RDMA contention, and separate confirmed findings from working hypotheses when talking to customers.


• Configure and validate RoCEv2 congestion control (ECN/DCQCN, PFC, DCBX) and reconcile host-side NIC settings against the switch fabric's actual behavior.


• Diagnose GPU hardware faults using DCGM telemetry and Xid error codes, distinguishing genuine hardware failures (RMA) from driver, firmware, or configuration issues.


• Own driver and firmware version compatibility across the fleet: GPU driver, NIC firmware (OFED/MOFED), kernel, and validate compatibility before and after updates.


• Provision, re-provision, and decommission GPU nodes: power-cycling, secure wipe, re-imaging, BIOS/NUMA/PCIe topology validation, and acceptance testing before handoff to a customer.


• Triage edge-network and jumpbox access issues affecting customer provisioning workflows (e.g., automated blocking of legitimate high-concurrency SSH/Ansible traffic).


• Write clear incident updates and post-resolution summaries for customer engineering teams, and feed recurring issues back into provisioning defaults and acceptance-test procedures.

Requirements

• Hands-on experience with RDMA fabrics — RoCEv2 specifically, including ECN/DCQCN, PFC, and DCBX trust-mode configuration on Mellanox/NVIDIA ConnectX NICs.


• Experience debugging multi-node GPU interconnect performance with NCCL and RDMA benchmarking tools (perftest, nccl-tests).


• Working knowledge of GPU interconnect topology (NVLink/NVSwitch, NUMA/PCIe affinity) and how it affects distributed training performance.


• Experience reading GPU health telemetry (NVIDIA DCGM, nvidia-smi, Xid codes) and managing driver/firmware compatibility across a fleet.


• Strong Linux systems administration, including SSH/host security tooling (fail2ban, sshd) and fleet automation (Ansible or equivalent).


• Comfortable owning live, customer-facing escalations on production systems, with clear written and verbal communication under time pressure.

Preferred Qualifications

• Familiarity with native InfiniBand fabrics (OpenSM/UFM) in addition to RoCEv2.


• Experience with scheduler-level network integration (Slurm, Kubernetes, Run:AI), including SR-IOV and multi-NIC bonding.


• Prior experience in a vendor-facing escalation role (e.g., NVIDIA/Mellanox support) or as a Field Applications Engineer.

What We Offer: Competitive base and strategic ownership in what we believe will be one of the largest infrastructure build-outs of the decade.

Compensation Range: $175,000 - $260,000

Rune is an equal opportunity employer. We celebrate diversity and are committed to creating an inclusive environment for all employees.