GPU Infrastructure Engineer

Polymer; Mountain View (CA); On-site; USD 175,000 - 260,000; Full time

Job summary

Rune in Mountain View is seeking a hands-on GPU Infrastructure Engineer to own customer-facing technical support for GPU clusters, including networking, provisioning, and health checks on live systems.

You will diagnose issues, reproduce and resolve or route them, and collaborate with customers to ensure peak performance, with emphasis on RoCEv2, NCCL, and GPU telemetry, while maintaining security and fleet stability.

Qualifications

• Hands-on RDMA fabrics setup and verification.
• Experience debugging multi-node GPU interconnect performance.
• Fluent Linux admin with fleet automation.
• Experience with SSH security tooling.
• Ability to own live customer escalations.

Responsibilities

• Serve as the primary technical responder for networking, GPU interconnect, and node issues on live clusters.
• Diagnose multi-node NCCL and RDMA performance regressions, including issues sourced to GPUDirect RDMA contention.
• Configure and validate RoCEv2 congestion control (ECN/DCQCN, PFC, DCBX) and reconcile host-side NIC settings against switch fabric behavior.
• Diagnose GPU hardware faults using DCGM telemetry and Xid error codes, distinguishing hardware failures from driver/config issues.
• Own driver and firmware version compatibility across the fleet: GPU driver, NIC firmware, kernel, and verify compatibility before/after updates.
• Provision, re-provision, and decommission GPU nodes: power-cycling, secure wipe, re-imaging, topology validation, and acceptance testing.
• Triage edge-network and jumpbox access issues affecting provisioning workflows.
• Write incident updates and post-resolution summaries for customer engineering teams.

Skills

RDMA RoCEv2

NCCL debugging

Linux admin

Ansible automation

GPU telemetry

Driver management

Tools

perftest

nccl-tests

Job description

GPU Infrastructure Engineer

Location: Mountain View, CA (not open to remote)

About Rune:

Every solar and wind power plant is a latent data center. Rune connects wasted power to the compute that needs it. Solar plants generate electricity the grid can't take: clipped by inverters, curtailed by operators, gone before it ever reaches a meter. Meanwhile, AI needs more power than new construction can deliver in time.

Rune deploys hardware, software, and financial solutions to put this power to work. We build modular micro data centers that install directly at solar sites, bypassing the traditional grid entirely and turning stranded power into compute. We're backed by Spark Capital, Union Square Ventures, and Lowercarbon Capital.

The Role:

We're looking for a hands-on engineer to own customer-facing technical support for our GPU clusters: networking, node provisioning, and GPU health. You'll be the primary responder to ensure our customers’ clusters are performing as expected, including diagnosing issues, resolving them, or routing them correctly, without needing to hand off the technical judgment to someone else.

Key Responsibilities:

• Serve as the primary technical responder for customer-reported networking, GPU-interconnect, and node issues on live clusters.
• Diagnose multi-node NCCL and RDMA performance regressions, including issues sourced to GPUDirect RDMA contention, and separate confirmed findings from working hypotheses when talking to customers.
• Configure and validate RoCEv2 congestion control (ECN/DCQCN, PFC, DCBX) and reconcile host-side NIC settings against the switch fabric's actual behavior.
• Diagnose GPU hardware faults using DCGM telemetry and Xid error codes, distinguishing genuine hardware failures (RMA) from driver, firmware, or configuration issues.
• Own driver and firmware version compatibility across the fleet: GPU driver, NIC firmware (OFED/MOFED), kernel, and validate compatibility before and after updates.
• Provision, re-provision, and decommission GPU nodes: power-cycling, secure wipe, re-imaging, BIOS/NUMA/PCIe topology validation, and acceptance testing before handoff to a customer.
• Triage edge-network and jumpbox access issues affecting customer provisioning workflows (e.g., automated blocking of legitimate high-concurrency SSH/Ansible traffic).
• Write clear incident updates and post-resolution summaries for customer engineering teams, and feed recurring issues back into provisioning defaults and acceptance-test procedures.

Requirements:

• Hands-on experience with RDMA fabrics — RoCEv2 specifically, including ECN/DCQCN, PFC, and DCBX trust-mode configuration on Mellanox/NVIDIA ConnectX NICs.
• Experience debugging multi-node GPU interconnect performance with NCCL and RDMA benchmarking tools (perftest, nccl-tests).
• Working knowledge of GPU interconnect topology (NVLink/NVSwitch, NUMA/PCIe affinity) and how it affects distributed training performance.
• Experience reading GPU health telemetry (NVIDIA DCGM, nvidia-smi, Xid codes) and managing driver/firmware compatibility across a fleet.
• Strong Linux systems administration, including SSH/host security tooling (fail2ban, sshd) and fleet automation (Ansible or equivalent).
• Comfortable owning live, customer-facing escalations on production systems, with clear written and verbal communication under time pressure.

Preferred Qualifications:

• Familiarity with native InfiniBand fabrics (OpenSM/UFM) in addition to RoCEv2.
• Experience with scheduler-level network integration (Slurm, Kubernetes, Run:AI), including SR-IOV and multi-NIC bonding.
• Prior experience in a vendor-facing escalation role (e.g., NVIDIA/Mellanox support) or as a Field Applications Engineer.

What We Offer:

Competitive base and strategic ownership in what we believe will be one of the largest infrastructure build-outs of the decade.

Compensation Range:

$175,000 - $260,000

Rune is an equal opportunity employer. We celebrate diversity and are committed to creating an inclusive environment for all employees.