GPU Cluster Reliability Engineer

Polymer; Mountain View (CA); On-site; USD 175,000 - 260,000; Full time

Job summary

Rune in Mountain View is seeking a hands-on GPU Infrastructure Engineer to own customer-facing technical support for GPU clusters, including networking, provisioning, and health checks on live systems.

You will diagnose issues, reproduce and resolve or route them, and collaborate with customers to ensure peak performance, with emphasis on RoCEv2, NCCL, and GPU telemetry, while maintaining security and fleet stability.

Qualifications

• Hands-on RDMA fabrics setup and verification.
• Experience debugging multi-node GPU interconnect performance.
• Fluent Linux admin with fleet automation.
• Experience with SSH security tooling.
• Ability to own live customer escalations.

Responsibilities

• Serve as the primary technical responder for networking, GPU interconnect, and node issues on live clusters.
• Diagnose multi-node NCCL and RDMA performance regressions, including issues sourced to GPUDirect RDMA contention.
• Configure and validate RoCEv2 congestion control (ECN/DCQCN, PFC, DCBX) and reconcile host-side NIC settings against switch fabric behavior.
• Diagnose GPU hardware faults using DCGM telemetry and Xid error codes, distinguishing hardware failures from driver/config issues.
• Own driver and firmware version compatibility across the fleet: GPU driver, NIC firmware, kernel, and verify compatibility before/after updates.
• Provision, re-provision, and decommission GPU nodes: power-cycling, secure wipe, re-imaging, topology validation, and acceptance testing.
• Triage edge-network and jumpbox access issues affecting provisioning workflows.
• Write incident updates and post-resolution summaries for customer engineering teams.

Skills

RDMA RoCEv2

NCCL debugging

Linux admin

Ansible automation

GPU telemetry

Driver management

Tools

perftest

nccl-tests

Job description

Rune in Mountain View is seeking a hands-on GPU Infrastructure Engineer to own customer-facing technical support for GPU clusters, including networking, provisioning, and health checks on live systems.

You will diagnose issues, reproduce and resolve or route them, and collaborate with customers to ensure peak performance, with emphasis on RoCEv2, NCCL, and GPU telemetry, while maintaining security and fleet stability.