Remote Forward-Deployed SRE for AI Infra

Andromeda Cluster; United States; Hybrid; USD 180,000 - 240,000; Full time

Job summary

Andromeda Cluster is seeking a Forward Deployed Engineer - SRE to work inside customer environments, optimizing large-scale GPU training and inference pipelines. You will onboard teams, tune runs, and debug failures, owning reliability of the platform and high-performance interconnects.

You will collaborate with customers to diagnose failures, reproduce issues, and ship fixes that improve the platform. This role blends hands-on ops with customer-facing engineering in a high-growth AI infra

Qualifications

• Hands-on experience operating GPU clusters in production.
• Production experience with InfiniBand, RoCE, or NVLink fabrics.
• Understanding GPU memory hierarchies, ECC behavior, and failure modes from direct experience.
• Production-grade Kubernetes with GPU workloads experience.
• Strong programming in Python, Go, or Bash.
• Infra-as-Code (Terraform, Helm, Ansible).
• Experience leading incident response for distributed systems.
• Ability to explain findings to a customer team without condescension.

Responsibilities

• Own onboarding end to end for teams running large-scale training/inference workloads.
• Diagnose real failures in customer environments: NCCL timeouts, I/O stalls, degraded links.
• Profile and improve distributed training performance on live workloads.
• Own reliability outcomes for the accounts you’re deployed on.
• Ensure health of high-speed interconnects (InfiniBand, RoCE, NVLink).
• Build monitoring for GPU telemetry and dashboards.
• Turn repeated deployments into automation: provisioning, health checks, preflight validation.
• Lead incident response and postmortem with systemic fixes.

Skills

GPU clusters

Kubernetes

Slurm

NVIDIA drivers

CUDA toolkit

Python

Go

Terraform

Tools

NVIDIA driver management

Container runtimes

DCGM / nvidia-smi

Job description

Andromeda Cluster is seeking a Forward Deployed Engineer - SRE to work inside customer environments, optimizing large-scale GPU training and inference pipelines. You will onboard teams, tune runs, and debug failures, owning reliability of the platform and high-performance interconnects.

You will collaborate with customers to diagnose failures, reproduce issues, and ship fixes that improve the platform. This role blends hands-on ops with customer-facing engineering in a high-growth AI infra