Remote Customer Reliability Engineer - AI Compute Platform

Andromeda; United States; Remote; USD 120,000 - 180,000; Full time

Job summary

Andromeda is seeking a Customer Reliability Engineer to manage large AI training and inference workloads on GPU clusters. You’ll triage, diagnose, and resolve Linux/Kubernetes issues, work with providers to identify fault sources, and build monitoring and tooling to prevent incidents.

Strong Linux skills and scripting ability are essential in a remote, on-call role. This position emphasizes incident response, collaboration with providers, and developing runbooks and automation to improve

Qualifications

• Real Linux troubleshooting ability from the command line.
• Working knowledge of Kubernetes: pods, nodes, deployments, services, scheduling, and investigation.
• Can write a script in Python or Bash to automate something repetitive.
• Strong writing. Explain a technical problem to a frustrated customer clearly.
• Willing to join a 24/7 on-call rotation.

Responsibilities

• Triage and fix customer issues: reproduce, diagnose, fix or elevate, close the loop.
• Debug at the Linux layer: processes, networking, storage, kernel logs, resource contention, systemd, journald.
• Dig into Kubernetes problems like pods stuck pending or crash-looping, node conditions, scheduling failures, resource limits.
• Work GPU failures: driver and device-plugin issues, XID errors, thermal throttling, nodes that need cordoning or draining, jobs failing across multiple nodes.
• Escalate when you're past your depth, with the evidence already gathered.
• Take part in a 24/7 on-call rotation.
• First response on alerts and customer-reported outages: assess impact, set severity, pull in the right people.
• Keep customers updated during incidents. Clear status, honest unknowns, no silence.
• Write up what happened, then turn it into a runbook, an alert, or a fix so it costs less next time.
• Work out whether a fault is provider-side, ours, or the customer's before it gets handed anywhere.
• Open tickets with compute providers and chase them down rather than waiting.
• Track recurring provider failures and flag the patterns to the people making sourcing decisions.
• Write Python or Bash to automate the checks you'd otherwise run by hand.
• Build and improve monitoring: cluster and node health checks, GPU telemetry, dashboards, alerts that fire on real problems.
• Keep runbooks and customer docs current as you go.

Skills

Linux troubleshooting

Kubernetes

Python/Bash scripting

Strong writing

24/7 on-call rotation

Tools

Prometheus

Grafana

Datadog

Job description

Andromeda is seeking a Customer Reliability Engineer to manage large AI training and inference workloads on GPU clusters. You’ll triage, diagnose, and resolve Linux/Kubernetes issues, work with providers to identify fault sources, and build monitoring and tooling to prevent incidents.

Strong Linux skills and scripting ability are essential in a remote, on-call role. This position emphasizes incident response, collaboration with providers, and developing runbooks and automation to improve