Senior Site Reliability Engineer

REALM; Brussel; Remote; EUR 125,000 - 179,000; Full time

Job summary

REALM is seeking a Senior Site Reliability Engineer to own GPU infrastructure for training and inference workloads across multiple providers and regions. You will diagnose issues from the network fabric through the kernel to the ML framework and drive reliability improvements.

You’ll work remotely with international teams, maintain InfiniBand/NVLink health, and build automation for provisioning, monitoring, and incident response. Strong Linux, CUDA, and Kubernetes skills are essential.

Qualifications

• Experience running large GPU clusters in production.
• Production experience with InfiniBand, RoCE or NVLink.
• Understanding of training job behavior at the systems level (NCCL, CUDA, PyTorch distributed).
• Strong Linux, kernel tuning and NVIDIA driver/CUDA management.
• Kubernetes with GPU workloads or Slurm/HPC schedulers.
• Python/Go/Bash tooling; Terraform/Helm/Ansible.

Responsibilities

• Designing and running multi-provider, multi-region GPU clusters for large training jobs.
• Acting as the technical contact for customers, from onboarding through to live troubleshooting and tuning.
• SLOs, capacity planning and reliability across a mixed GPU fleet.
• Keeping InfiniBand, RoCE and NVLink fabrics healthy.
• Monitoring that shows GPU utilisation, interconnect throughput and hardware health.
• Automation for provisioning, health checks, self-healing and driver updates.
• Leading incidents where the root cause could sit in hardware, network, orchestration or the ML stack.

Skills

GPU cluster management

Linux kernel tuning

SRE incident leadership

Python/Go/Bash scripting

Tools

InfiniBand

RoCE

NVLink

CUDA

NCCL

nvidia-smi

DCGM

Terraform

Helm

Ansible

Kubernetes

Slurm

Job description

Senior Site Reliability Engineer, GPU Infrastructure

Remote, EMEA or APAC | Full-time | Competitive base + meaningful equity

The company

I’m hiring for an AI infrastructure company that sources GPU compute from dozens of providers and runs it as one managed platform for AI labs and model builders. They have tens of thousands of GPUs under management and a team of around 50. They’re profitable and backed by some of the best-known names in AI. It’s a confidential search, so I can’t name them here, but I’ll tell you everything on a call.

The role

This isn’t a generalist SRE seat. You’ll run large GPU clusters for distributed training and inference, spread across multiple providers and regions, and work directly with the customers pushing them hardest. When a training run slows down, you’ll be the one tracing it from the network fabric through the kernel to the framework.

What you’ll own

• Designing and running multi-provider, multi-region GPU clusters for large training jobs
• Acting as the technical contact for customers, from onboarding through to live troubleshooting and tuning
• SLOs, capacity planning and reliability across a mixed GPU fleet
• Keeping InfiniBand, RoCE and NVLink fabrics healthy
• Monitoring that shows what’s happening with GPU utilisation, interconnect throughput and hardware health
• Automation for provisioning, health checks, self-healing and driver and firmware updates
• Leading incidents where the root cause could sit in hardware, network, orchestration or the ML stack

What you’ll need

• You’ve run large GPU clusters (A100, H100, B200 or similar) in production yourself
• Production experience with InfiniBand, RoCE or NVLink for distributed training
• A good understanding of how big training jobs behave at the systems level: NCCL, CUDA, PyTorch distributed or similar
• Strong Linux, including kernel tuning and NVIDIA driver and CUDA management
• Kubernetes with GPU workloads, or Slurm and other HPC schedulers (either counts)
• Python, Go or Bash for building proper tooling, plus Terraform, Helm, Ansible or similar
• GPU monitoring with DCGM and nvidia-smi
• Experience leading messy incidents where the cause could be anywhere from firmware to application code

Nice to have

• Parallel file systems like VAST, WEKA, Lustre or GPFS
• Profiling and tuning distributed training performance
• Physical cluster design: racks, power, cooling, network topology and burn-in