Senior Site Reliability Engineer

andromeda; United States; Remote; USD 180,000 - 280,000; Full time

Job summary

andromeda is seeking a senior infrastructure engineer to operate and improve large, distributed GPU clusters for AI training and inference workloads. You will own performance tracing across hardware, network fabric, orchestration, and ML software while collaborating with customers and compute providers.

The role emphasizes design of multi-region GPUs, onboarding customers, reliability objectives, capacity planning, and incident leadership with a global remote setup.

Qualifications

• Direct production experience operating large GPU clusters and understanding hardware failure modes.
• Experience with high-performance networking for distributed training and diagnosing degraded links.
• Strong knowledge of distributed training systems (NCCL, CUDA, PyTorch distributed, DeepSpeed, Megatron, FSDP).

Responsibilities

• Design multi-provider, multi-region GPU clusters for throughput and cost.
• Onboard customers and serve as technical POC for performance issues.
• Set reliability objectives and plan capacity across hardware fleets.
• Monitor high-speed interconnects (InfiniBand, RoCE, NVLink) performance.
• Build GPU observability for utilization, memory pressure, fabric throughput, health.
• Develop automation for provisioning, health checks, scheduling, and incident response.

Skills

GPU cluster ops

Distributed training

Kubernetes ops

Incident leadership

Capacity planning

Tools

NCCL

CUDA

PyTorch Distributed

DeepSpeed

Megatron

FSDP

Terraform

Helm

Ansible

Job description

Role overview

Operate and improve large, distributed GPU clusters that support demanding AI training and inference workloads. This senior infrastructure role combines systems engineering, customer support, capacity planning, and incident leadership, with performance traced across hardware, network fabric, orchestration, and ML software.

Responsibilities

• Design multi-provider, multi-region GPU clusters, making topology, scheduling, networking, and storage choices that affect training throughput and cost.
• Onboard customers running large training jobs, serve as a technical point of contact, and resolve performance or reliability problems with them.
• Set reliability objectives and error budgets for GPU-specific failures; plan capacity across varied hardware fleets.
• Monitor and troubleshoot high-speed interconnects such as InfiniBand, RoCE, and NVLink, including issues that slow distributed communication.
• Build GPU-focused observability for utilization, memory pressure, fabric throughput, job performance, and hardware health.
• Develop automation for provisioning, health checks, scheduling, self-recovery, and driver or firmware lifecycle management; lead incident response and follow-up improvements.

Requirements

• Direct production experience operating large GPU clusters, such as NVIDIA A100, H100, B200, or equivalent, and understanding their hardware failure modes.
• Production experience with high-performance networking for distributed training, including diagnosing degraded links, congestion, or slow collective operations.
• Working knowledge of distributed training systems such as NCCL, CUDA, PyTorch distributed, DeepSpeed, Megatron, or FSDP.
• Strong Kubernetes operations experience with GPU workloads; familiarity with device plugins, topology-aware scheduling, multi-cluster setups, or custom operators is relevant.
• Software engineering ability in Python, Go, or Bash, plus infrastructure-as-code experience with tools such as Terraform, Helm, or Ansible.
• Experience building GPU-specific monitoring and leading incidents across hardware, firmware, network, drivers, orchestration, and applications.

Nice to have

• Parallel storage experience, training performance optimization, or hands-on cluster buildout involving rack, power, cooling, network design, or hardware validation.
• Experience mentoring or leading infrastructure engineers.

Benefits and work setup

The position is listed as global remote or San Francisco. It offers substantial autonomy to shape infrastructure and technical direction while working with customers and compute providers.