Staff Infra Engineer – Global Inference & Kubernetes

Xpertise Recruitment

San Francisco (CA)

On-site

USD 350,000 - 450,000

Full time

5 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Xpertise Recruitment is representing an on-site Staff Infrastructure Engineer in San Francisco to own the production serving layer for managed inference. The role involves designing, operating, and scaling Kubernetes across AWS and GCP, and debugging NVIDIA GPU systems on real hardware and traffic.

You will collaborate with model teams, extend deployment tooling for Voyager across NVIDIA/Trainium/TPU, and build robust observability and failover mechanisms to ensure low latency and high

Qualifications

  • Five or more years of infrastructure engineering, currently operating at staff level or above in an individual contributor role.
  • Hands-on experience operating production Kubernetes at scale across AWS and GCP.
  • Direct, day-to-day experience with inference infrastructure.
  • Hands-on experience with NVIDIA GPU systems: drivers, CUDA, NCCL, and node health.
  • Hands-on experience with an inference serving framework such as vLLM, SGLang, TensorRT-LLM, or Dynamo.
  • Experience with HPC networking, including InfiniBand, RDMA/EFA, or MPI.
  • Experience with observability and reliability for latency-sensitive services, including metrics, alerting, and automated failover.
  • A track record of building highly available, multi-cloud infrastructure.
  • Experience deploying more than one accelerator family, such as Trainium, TPU, or a mixed fleet.
  • Previous experience at Meta.

Responsibilities

  • Design, operate, and scale Kubernetes across AWS, GCP, and GPU cloud providers, supporting global inference demand at 1,000+ tokens per second.
  • Own the production serving layer for LLM inference.
  • Debug NVIDIA GPU systems end to end, including drivers, CUDA, NCCL, node health, and the network fabric.
  • Work with the model team so performance work holds up in production, on real hardware and real traffic.
  • Extend the deployment tooling so a model can be shipped the same way onto NVIDIA, Trainium, and TPU hosts, including mixed fleets.
  • Build observability for time to first token, inter-token latency, throughput, and availability, measured per model, per chip, and per region.
  • Own reliability engineering: alerting, automated failover, self-healing infrastructure, and traffic routing across models, chips, and regions.

Skills

Kubernetes at scale
AWS
GCP
NVIDIA GPU expertise
Inference infrastructure
Observability & reliability
Multi-cloud infra
Production systems

Tools

vLLM
TensorRT-LLM
Dynamo
NCCL

Job description

Xpertise Recruitment is representing an on-site Staff Infrastructure Engineer in San Francisco to own the production serving layer for managed inference. The role involves designing, operating, and scaling Kubernetes across AWS and GCP, and debugging NVIDIA GPU systems on real hardware and traffic.

You will collaborate with model teams, extend deployment tooling for Voyager across NVIDIA/Trainium/TPU, and build robust observability and failover mechanisms to ensure low latency and high

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Inference Engineer, Global GPU Deployment & Kubernetes
Inference Engineer, Global GPU Deployment & Kubernetes

Hyperbolic • San Francisco (CA)

On-site
USD 140,000 - 230,000
Senior Infrastructure Engineer
Senior Infrastructure Engineer

Xpertise Recruitment • San Francisco (CA)

On-site
USD 350,000 - 450,000
Staff Infra Engineer - Global GPU ML Inference
Staff Infra Engineer - Global GPU ML Inference

The Token Company • San Francisco (CA)

On-site
USD 120,000 - 160,000
Equity
Housing
Food
+2
Senior Staff AI/ML Inference Engineer (Kubernetes & GPU)
Senior Staff AI/ML Inference Engineer (Kubernetes & GPU)

ServiceNow • Kirkland (WA)

On-site
USD 201,000 - 352,000
Health plans
401(k) Match
ESPP
+3
Senior Staff AI/ML Inference Engineer - GPU, Kubernetes
Senior Staff AI/ML Inference Engineer - GPU, Kubernetes

ServiceNow • Santa Clara (CA)

Hybrid
USD 201,000 - 352,000
Health plans
401(k) Plan with company match
ESP P (Employee Stock Purchase Plan)
+3
Senior AI/ML Inference Engineer — GPU & LLMs on Kubernetes
Senior AI/ML Inference Engineer — GPU & LLMs on Kubernetes

ServiceNow • San Diego (CA)

Hybrid
USD 181,000 - 317,000
Kubernetes Platform Engineer for AI Inference
Kubernetes Platform Engineer for AI Inference

present of AI • San Francisco (CA)

On-site
USD 180,000 - 240,000
Staff Infra Engineer — Scale Global Kubernetes & AI Infra
Staff Infra Engineer — Scale Global Kubernetes & AI Infra

Ivo • San Francisco (CA)

On-site
USD 287,000 - 485,000
Competitive pay
Equity
Relocation assistance
+7
Global Inference Engineer — End-to-End Deployment
Global Inference Engineer — End-to-End Deployment

Hyperbolic • San Francisco (CA)

On-site
USD 150,000 - 190,000
AI Infra Engineer – SRE (Kubernetes)
AI Infra Engineer – SRE (Kubernetes)

Berrybytes • United States

On-site
USD 110,000 - 150,000