Senior AI Infra Engineer: GPU Scale & Resilience (Hybrid)

Calance; Costa Mesa (CA); Hybrid; USD 180,000 - 240,000; Full time

Job summary

Calance seeks a Senior AI Infrastructure Engineer to own the scalable GPU training pipeline, from bring-up to operation. You will design fault-tolerant, self-healing GPU clusters, tune high-speed interconnects, and automate deployment with IaC.

You will enhance NCCL, Run:AI, and Ray scheduling, ensuring high availability for ML workloads across teams. You will lead hardware bring-up, firmware management, and fabric optimization while maintaining fault isolation and observability in a hybrid

Qualifications

• Hands-on experience in infrastructure/HPC/datacenter engineering for GPU compute at scale.
• Experience with high-performance interconnects (NVLink, InfiniBand, RoCE) in large GPU clusters.
• Experience with Kubernetes and GPU scheduling (Run:AI or similar) is highly preferred.

Responsibilities

• Rack, stack, cabling, and bring up GPU compute nodes including topology, power, cooling, BIOS, and burn-in validation.
• Build and tune interconnect fabrics (NVLink, InfiniBand, RoCE, Spectrum-X) for large GPU clusters.
• Automate cluster deployment/configuration end-to-end with infrastructure as code; manage firmware/driver.
• Operate and extend Kubernetes/Run:AI for GPU scheduling and multi-tenant isolation.
• Own fleet health: monitoring, alerting, and rapid triage of hardware/network faults.
• Onboard researchers/engineers; assist debugging/training workloads when infrastructure is the bottleneck.
• Collaborate with product teams to translate compute needs into platform capabilities.

Skills

Automation

Cluster management

Hardware fault diagnosis

High performance computing

Tools

Kubernetes

Run:AI

Ray

NCCL

NVLink

InfiniBand

VAST/NDN/Weka

Job description

Calance seeks a Senior AI Infrastructure Engineer to own the scalable GPU training pipeline, from bring-up to operation. You will design fault-tolerant, self-healing GPU clusters, tune high-speed interconnects, and automate deployment with IaC.

You will enhance NCCL, Run:AI, and Ray scheduling, ensuring high availability for ML workloads across teams. You will lead hardware bring-up, firmware management, and fabric optimization while maintaining fault isolation and observability in a hybrid