Reliability Engineer

REALM; United States; Remote; USD 120,000 - 190,000; Full time

Job summary

REALM is seeking a Customer Reliability Engineer to own issue resolution for AI infrastructure. You’ll diagnose failures in training jobs, nodes, and GPUs, working across Linux, Kubernetes, and cloud providers.

You’ll build tooling and monitoring to prevent reoccurrence and participate in a 24/7 on‑call rotation. You’ll write Python or Bash scripts, communicate clearly with customers, and help determine whether faults lie with the provider, platform, or customer.

Qualifications

• Hands‑on Linux troubleshooting: analyze logs, processes, networking, and disks without a script.
• Working knowledge of Kubernetes: pods, nodes, deployments, services, scheduling, and failure investigation.
• Ability to write Python or Bash scripts to automate repetitive work.
• Excellent written communication: explain technical problems clearly to customers.
• Sound judgement under pressure: know what to check first and how to keep stakeholders updated.

Responsibilities

• Investigate and resolve customer issues from first report to resolution; reproduce, diagnose, fix or escalate, and follow up with the customer.
• Troubleshoot at the Linux level: processes, networking, storage, kernel logs, resource contention, systemd and journald.
• Investigate Kubernetes problems such as pending or crash‑looping pods, node conditions, scheduling failures and resource limits.
• Learn to handle GPU‑related failures: drivers, device plugins, XID errors, thermal throttling, cordoning and draining.
• Join a 24/7 on‑call rotation and act as first responder on alerts and outages; assess impact and bring in the right people.
• Write up incidents and turn lessons into runbooks, alerts or fixes.
• Determine fault sits with provider, platform or customer; raise tickets and follow through to resolution.
• Spot recurring provider issues and share patterns with sourcing decisions.
• Write Python or Bash scripts to automate checks and build monitoring.
• Build and improve monitoring and dashboards to catch real problems; keep runbooks updated.

Skills

Linux troubleshooting

Kubernetes knowledge

Python or Bash scripting

Written communication

Decision making under pressure

Tools

Prometheus

Grafana

Datadog

Terraform

Ansible

Helm

Job description

Customer Reliability Engineer - AI Infrastructure

Remote | Full-time

About the company

Our client is a fast-growing, profitable AI infrastructure company that sits between the businesses that own GPU capacity and the AI teams who need it. They source compute from data centres around the world, test and certify it, and hand it to customers as a single managed cluster with one contract and one point of support. Their customers include AI labs and startups running large training and inference workloads.

The team is small, hands‑on and growing quickly, which means real ownership and the chance to shape how reliability works from early on.

The role:

When a customer's training job fails hours into a run, or a node drops offline, you're the person who works out what went wrong and gets it fixed. Sometimes the cause is the customer's setup, sometimes it's the platform, and often it's the underlying provider. Your job is to figure out which, fix it or get the right people fixing it, and then build the checks that stop it happening again.

You’ll need to be comfortable on the Linux command line and understand how Kubernetes works. You don't need prior GPU or HPC experience. That's learned on the job.

What you'll do:

Investigate and resolve customer issues

• Take ownership of issues from first report to resolution: reproduce, diagnose, fix or escalation, and follow up with the customer
• Troubleshoot at the Linux level, including processes, networking, storage, kernel logs, resource contention, systemd and journald
• Investigate Kubernetes problems such as pending or crash-looping pods, node conditions, scheduling failures and resource limits
• Learn to handle GPU‑related failures, including driver and device-plugin issues, XID errors, thermal throttling, cordoning and draining nodes, and multi‑node job failures
• Know when to escalation, and do it with the evidence already gathered

Respond to incidents

• Join a 24/7 on‑call rotation
• Act as first responder on alerts and customer‑reported outages: assess the impact, set severity and bring in the right people
• Keep customers informed throughout, with clear updates and honesty about what's still unknown
• Write up incidents afterwards and turn the lessons into runbooks, alerts or fixes

Work with compute providers

• Determine whether a fault sits with the provider, the platform or the customer before passing it on
• Raise tickets with providers and actively follow them through to resolution
• Spot recurring provider issues and share the patterns with the team responsible for sourcing decisions

Build tooling and monitoring

• Write Python or Bash scripts to automate checks you'd otherwise do manually
• Build and improve monitoring, including cluster and node health checks, GPU telemetry, dashboards and alerts that catch real problems
• Keep runbooks and customer documentation up to date

What you'll bring:

• Hands‑on Linux troubleshooting skills: you can work through logs, processes, networking and disk without a script to follow
• Working knowledge of Kubernetes (pods, nodes, deployments, services, scheduling) and how to investigate when something breaks
• The ability to write Python or Bash scripts to automate repetitive work
• Excellent written communication: you can explain a technical problem to a frustrated customer clearly and respectfully
• Sound judgement under pressure: knowing what to check first, when to raise and how to keep people updated while you work

Nice to have:

• High‑performance networking experience (InfiniBand, RoCE) or NCCL
• NVIDIA GPUs in production: drivers, CUDA, DCGM, the Kubernetes device plugin
• HPC or batch schedulers such as Slurm
• A previous customer‑facing technical role, such as support engineering, technical account management, solutions or professional services
• Prometheus, Grafana, Datadog or similar
• Infrastructure as code: Terraform, Ansible or Helm
• A genuine interest in AI infrastructure and how large training jobs behave

What's on offer:

• Early‑stage impact at a profitable, fast-growing company at the centre of the AI infrastructure market
• Competitive salary [range - confirm] plus meaningful equity
• Comprehensive benefits including healthcare, dental and vision for you and your dependants, 401(k) and unlimited holiday (US); local benefits for employees elsewhere
• The chance to learn GPU and AI infrastructure from the inside