Staff Site Reliability Engineer - AI Infrastructure

Hamilton Barnes Associates Limited; San Francisco (CA); On-site; USD 297,500 - 402,500; Full time

Benefits offered by this job

Huge stock options

Company bonus

Unlimited PTO

401K + 4% match

Job summary

Hamilton Barnes Associates Limited is seeking a Staff Site Reliability Engineer to lead reliability across large-scale GPU infrastructure used for AI training and inference. You will drive incident response, performance tuning, and platform health, collaborating with cross-functional teams to scale operations.

The role emphasizes hands-on GPU hardware/software expertise, Kubernetes for GPU clusters, and strong programming in Go, Python, or Rust.

Qualifications

• Hands-on experience operating large-scale GPU infrastructure environments.
• Staff-level SRE or infrastructure engineering experience supporting mission-critical production systems.
• Deep expertise with NVIDIA GPU platforms including H100, H200, B200, or GB200 systems.
• Strong software engineering skills in Go, Python, or Rust.

Responsibilities

• Lead high-priority incident response across distributed GPU infrastructure environments.
• Diagnose and resolve issues across the stack including PyTorch, NCCL, CUDA, drivers, networking fabrics, and hardware layers.
• Own day-to-day operational health of large-scale GPU fleets including lifecycle management and firmware upgrades.
• Build and maintain observability systems, GPU telemetry platforms, automation tooling, and health-check frameworks.
• Partner with infrastructure, product, and platform engineering teams to improve reliability and scalability.
• Mentor engineers across reliability engineering and incident management practices.
• Contribute to a long-term reliability strategy for hyperscale AI infrastructure.

Skills

GPU infra ops

Staff-level SRE

NVIDIA GPUs

CUDA/NCCL

Go

Python

Rust

Kubernetes

HPC schedulers

Linux systems

Incident response

Tools

Observability tooling

Job description

Looking for a role with plenty of growth opportunities?

Join a rapidly growing AI infrastructure provider delivering large-scale compute solutions for AI training and inference across global cloud and GPU environments. The organization partners with leading AI companies and infrastructure providers to build reliable, high-performance platforms supporting next-generation AI workloads.

This opportunity is for a Staff Site Reliability Engineer to lead the reliability of large-scale GPU infrastructure, covering deployment, GPU health, distributed training, networking, and incident response. Working as a senior technical leader, the role focuses on improving infrastructure reliability, operational standards, and platform performance across complex AI environments.

Ready to make a move? Get in touch and apply today!

Responsibilities

• Lead high-priority incident response across distributed GPU infrastructure environments
• Diagnose and resolve issues across the stack, including PyTorch, NCCL, CUDA, drivers, networking fabrics, and hardware layers
• Own day-to-day operational health of large-scale GPU fleets, including lifecycle management, validation, firmware rollouts, upgrades, and repair workflows
• Build and maintain observability systems, GPU telemetry platforms, automated remediation tooling, and health-check frameworks
• Define and scale operational practices, including on-call rotations, escalation processes, incident response, and postmortem standards
• Partner closely with infrastructure, product, and platform engineering teams to improve reliability and scalability
• Participate in customer-facing technical discussions, incident reviews, architecture workshops, and workload planning sessions
• Influence physical infrastructure design, including rack layouts, power density, cooling strategies, burn-in procedures, and network topology decisions
• Mentor engineers across reliability engineering, systems operations, and incident management practices
• Contribute to a long-term reliability strategy for hyperscale AI infrastructure environments

Skills/Must Have

• Multiple years of hands-on experience operating large-scale GPU infrastructure environments
• Proven Staff-level SRE or infrastructure engineering experience supporting mission-critical production systems
• Deep expertise with NVIDIA GPU platforms including H100, H200, B200, or GB200 systems
• Strong understanding of GPU memory hierarchy, ECC behaviour, NVLink, NVSwitch, thermal management, and hardware failure analysis
• Production experience with InfiniBand, RoCE, and high-performance distributed training fabrics
• Deep understanding of distributed AI training technologies, including NCCL, CUDA, PyTorch Distributed, FSDP, DeepSpeed, and Megatron
• Strong software engineering skills in Go, Python, or Rust
• Experience building production-grade automation, tooling, fleet management systems, or reliability platforms
• Hands-on experience with Kubernetes GPU environments, Slurm, or HPC schedulers
• Strong Linux systems expertise, including kernel tuning, CUDA lifecycle management, cgroups/namespaces, BPF/performance analysis, and firmware operations
• Calm, structured incident response capabilities within high-pressure production environments
• Ability to communicate effectively with highly technical customers, providers, and executive stakeholders

Desirable Skills

• Experience building custom GPU fleet health systems or fabric controllers
• Expertise with distributed storage systems such as VAST, Weka, Lustre, or GPFS
• Experience optimizing distributed training efficiency, checkpointing, and multi-thousand-GPU job performance
• Background supporting enterprise AI infrastructure customers in customer-facing technical roles
• Open-source contributions within the GPU, Kubernetes, or AI infrastructure ecosystem
• Public speaking, technical writing, or community leadership within AI infrastructure or HPC domain

Benefits

• Huge stock options
• Company bonus
• Unlimited PTO
• 401K + 4% match

Salary

• $350,000 base salary