Senior AI Cloud SRE — HPC & GPU Infrastructure

Neura Market; San Francisco, Northern (CA, KY); Hybrid; USD 180,000 - 240,000; Full time

Job summary

Lambda is seeking an experienced Site Reliability Engineer to monitor and optimize AI compute clusters. You will deploy, scale, and automate HPC environments, building reliable operations for GPU-rich workloads across on-site teams.

You will implement robust tooling in Python/Go, use Prometheus and Grafana for observability, and drive incident response with a focus on stability and efficiency. This role requires presence in SF or Bellevue with a hybrid setup.

Qualifications

• 7+ years of experience in Site Reliability Engineering, HPC Engineering, DevOps, or a similar role.
• Strong understanding of modern AI infrastructure, from GPU architectures to hardware performance optimization.
• Strong Linux skills in distributed environments.
• Experience configuring and troubleshooting InfiniBand (IB), RoCE, CLOS fabrics, 100GbE, Ethernet switching, GPU-direct, and NCCL.
• Proficiency in Python and Go with internal tooling improvements.
• Experience with monitoring and alerting tools (Prometheus, Grafana, Clickhouse).
• Proficiency with automation/configuration tools (Ansible, Terraform).
• Excellent problem-solving and attention to detail; continuous improvement mindset.

Responsibilities

• Build and operate monitoring and alerting for cluster health across fabric, GPU, power/thermal, and job signals.
• Remotely deploy and configure large-scale HPC clusters for AI workloads with automation.
• Automate cluster lifecycle: OS, firmware, drivers, networking as code (Ansible, Terraform).
• Create runbooks and automated remediations for common cluster failures for safe execution by Support teams.
• Troubleshoot cluster issues across InfiniBand/RoCE, NCCL, GPU-direct, fabric, and switches with on-site teams.
• Lead incident responses during on-call rotations.
• Contribute to SOPs and provide clear requirements to engineering for stability and efficiency.

Skills

SRE

HPC Engineering

DevOps

Python

Go

Monitoring

Tools

Prometheus

Grafana

Clickhouse

Job description

Lambda is seeking an experienced Site Reliability Engineer to monitor and optimize AI compute clusters. You will deploy, scale, and automate HPC environments, building reliable operations for GPU-rich workloads across on-site teams.

You will implement robust tooling in Python/Go, use Prometheus and Grafana for observability, and drive incident response with a focus on stability and efficiency. This role requires presence in SF or Bellevue with a hybrid setup.