Senior GPU Cluster Reliability Engineer — Remote

andromeda; United States; Remote; USD 180,000 - 280,000; Full time

Job summary

andromeda is seeking a senior infrastructure engineer to operate and improve large, distributed GPU clusters for AI training and inference workloads. You will own performance tracing across hardware, network fabric, orchestration, and ML software while collaborating with customers and compute providers.

The role emphasizes design of multi-region GPUs, onboarding customers, reliability objectives, capacity planning, and incident leadership with a global remote setup.

Qualifications

• Direct production experience operating large GPU clusters and understanding hardware failure modes.
• Experience with high-performance networking for distributed training and diagnosing degraded links.
• Strong knowledge of distributed training systems (NCCL, CUDA, PyTorch distributed, DeepSpeed, Megatron, FSDP).

Responsibilities

• Design multi-provider, multi-region GPU clusters for throughput and cost.
• Onboard customers and serve as technical POC for performance issues.
• Set reliability objectives and plan capacity across hardware fleets.
• Monitor high-speed interconnects (InfiniBand, RoCE, NVLink) performance.
• Build GPU observability for utilization, memory pressure, fabric throughput, health.
• Develop automation for provisioning, health checks, scheduling, and incident response.

Skills

GPU cluster ops

Distributed training

Kubernetes ops

Incident leadership

Capacity planning

Tools

NCCL

CUDA

PyTorch Distributed

DeepSpeed

Megatron

FSDP

Terraform

Helm

Ansible

Job description

andromeda is seeking a senior infrastructure engineer to operate and improve large, distributed GPU clusters for AI training and inference workloads. You will own performance tracing across hardware, network fabric, orchestration, and ML software while collaborating with customers and compute providers.

The role emphasizes design of multi-region GPUs, onboarding customers, reliability objectives, capacity planning, and incident leadership with a global remote setup.