Remote GPU HPC Infrastructure Engineer

Valor Capital Group; New York (NY); Remote; USD 150,000 - 210,000; Full time

Job summary

ElevenLabs is seeking an HPC Infrastructure Engineer to own and operate our GPU clusters. You’ll manage provisioning, scheduling, monitoring, and upgrades across bare-metal hardware and rented capacity to maximize training throughput.

You’ll automate health checks, build scalable storage, tune Slurm-like schedulers, and work with datacenter staff. This role offers unusual autonomy in a lean, fast-moving team.

Qualifications

• Experience running large-scale Linux server or GPU environments in production.
• Deep NVIDIA stack knowledge (drivers, CUDA, NCCL, DCGM) or equivalent systems expertise.
• Comfort with bare-metal environments, server hardware, and high-speed networking.
• Strong automation skills in Python and/or Bash; IaC experience with Ansible or Terraform.
• Able to analyze metrics/logs with PromQL to identify root causes.
• Enjoy ownership of end-to-end scope and being relied upon by peers.
• Willingness to handle datacenter trips when needed.

Responsibilities

• Operate and improve our GPU fleet end to end: provisioning, scheduling, monitoring, upgrades, capacity planning.
• Build automation to keep the fleet healthy without human intervention: node health checks, automated draining, remediation, burn-in pipelines.
• Own the stack beneath the training code: OS images, NVIDIA drivers, CUDA, container runtimes, NCCL, high-speed networking.
• Run and tune job scheduling so researchers get compute fairly and fast.
• Build and maintain high-performance storage for datasets and checkpoints.
• Hunt down performance problems and fix the class of problem, not just the instance.
• Evaluate rented GPU capacity: benchmark, validate, ensure SLAs.
• Hands-on hardware work: racking, cabling, coordinating with datacenter staff.
• Keep clusters secure by default: access control, network isolation, secrets.

Skills

Linux servers

NVIDIA stack

Bare-metal

Python

Bash

Ansible

Terraform

PromQL

Networking

GPU hardware

Tools

Slurm

NCCL

InfiniBand

Job description

ElevenLabs is seeking an HPC Infrastructure Engineer to own and operate our GPU clusters. You’ll manage provisioning, scheduling, monitoring, and upgrades across bare-metal hardware and rented capacity to maximize training throughput.

You’ll automate health checks, build scalable storage, tune Slurm-like schedulers, and work with datacenter staff. This role offers unusual autonomy in a lean, fast-moving team.