Senior GPU Infrastructure Engineer - Systems Integrator

Hamilton Barnes; United Kingdom; Remote; GBP 108,000 - 132,000; Full time

Benefits offered by this job

4-day work week

Remote-first with flexibility

Uncapped holiday

Competitive compensation

Job summary

Hamilton Barnes is seeking a GPU Platform Reliability Engineer to own reliability, performance, and automation for the AI/GPU platform. You will bridge hardware telemetry with cloud-native orchestration, acting as the escalation point for a live GPU fleet and building automated remediation tooling.

The role includes a 4-day week, remote flexibility, and a competitive package. Expertise in NVIDIA stacks, Kubernetes and IaC is essential for success.

Responsibilities

• Act as the top internal escalation point for hardware, PCIe, GPU memory (ECC) and XID driver faults across the fleet.
• Build and maintain automated health check frameworks using NVIDIA DCGM, NCCL test suites and telemetry agents.
• Own automated rollout pipelines for GPU BIOS, system BIOS, BMC firmware and CUDA drivers across live clusters with no unannounced customer impact.
• Diagnose bottlenecks, link degradation and optimize data paths across InfiniBand and RoCEv2 fabrics to prevent GPU starvation.
• Tune NVIDIA GPU Operator, OS and workload plugins; design node-cordon, eviction and repair workflows from telemetry.
• Drive full-stack IaC (Terraform, Ansible, Python) for fleet expansion and incident war-room resolution with blameless post-mortems.

Job description

Are you looking for an exciting new opportunity?

Join a sovereign cloud and AI platform that combines the openness of public cloud with the security and control of private environments, making the cloud easier to use, faster to deploy, and fairer to pay for. Offering public cloud, private cloud, and high-performance GPU infrastructure with clear, predictable pricing, it helps organisations innovate while keeping control of their data, infrastructure, and spend.

The organization is inviting applications from experienced professionals for the role of GPU Platform Reliability Engineer, acting as the technical authority for the reliability, performance, and automated orchestration of the AI and GPU platform layer. The ideal candidate will work at the boundary between server hardware telemetry, high-bandwidth interconnect fabrics, and cloud-native orchestration, serving as the highest internal escalation point for a live production GPU fleet and building automated remediation tooling. The role includes a 4-day working week and remote flexibility.

Responsibilities:

• Act as the top internal escalation point for persistent or intermittent hardware, PCIe, GPU memory (ECC) and XID driver faults across the GPU fleet, owning deep root-cause analysis
• Build and maintain automated pre-job and post-job health check frameworks using NVIDIA DCGM, NCCL test suites and custom telemetry agents
• Own automated rollout pipelines for GPU VBIOS, system BIOS, BMC firmware and CUDA driver stacks across live production clusters with zero unannounced customer impact
• Diagnose bottlenecks, packet drops and link degradation across InfiniBand (Quantum-2/X800) and RoCEv2 (Spectrum-X) fabrics, and optimise GPUDirect Storage and parallel file system (VAST Data) data paths to prevent GPU starvation
• Maintain and tune the NVIDIA GPU Operator, Network Operator, Kubeflow and Slurm plugins across bare-metal and containerised environments, and design automated node-cordoning, pod eviction and repair workflows driven by real-time telemetry
• Drive full-stack Infrastructure-as-Code (Terraform, Ansible, Python) for fleet expansion and region bring-ups, while leading war-room resolution of major incidents, managing NVIDIA and OEM vendor escalations and running blameless post-mortems

Skills/Must have:

• Experience 5+ years in SRE, Systems or Infrastructure Engineering supporting high-density, production-scale Linux environments, including 3+ years managing and troubleshooting data centre class NVIDIA GPUs (H100, B200 or newer) in distributed environments
• Deep expertise in the NVIDIA stack (CUDA runtime drivers, DCGM metrics, NVLink/NVSwitch topologies, XID error troubleshooting); advanced Kubernetes administration (CKA level, including CNI and Device Plugins) and Slurm
• InfiniBand subnet management, RoCEv2 lossless Ethernet (PFC, ECN) and NCCL tuning; strong Python, Go or Bash alongside Terraform and Ansible; Prometheus, Grafana and vector-based telemetry stacks for hardware observability
• Calm technical leadership under high-pressure incidents; excellent technical writing for post-mortems, runbooks and cross-team documentation

Desirable Skills:

• NVIDIA Certified Professional certifications (NCP-AII, NCP-IB)
• Certified Kubernetes Administrator (CKA) or Security Specialist (CKS)
• Experience with liquid-cooled rack architectures (Direct-to-Chip, CDU setups)

Benefits:

• 4-day working week (excluding event attendance)
• Uncapped holiday
• Remote-first working with flexibility and autonomy
• Competitive compensation and benefits package
• Inclusive, collaborative culture

Salary:

• £120k