Senior Infrastructure Engineer

Xpertise Recruitment; San Francisco (CA); Vor Ort; USD 350.000 - 450.000; Vollzeit

Zusammenfassung

Xpertise Recruitment is representing an on-site Staff Infrastructure Engineer in San Francisco to own the production serving layer for managed inference. The role involves designing, operating, and scaling Kubernetes across AWS and GCP, and debugging NVIDIA GPU systems on real hardware and traffic.

You will collaborate with model teams, extend deployment tooling for Voyager across NVIDIA/Trainium/TPU, and build robust observability and failover mechanisms to ensure low latency and high

Qualifikationen

• Five or more years of infrastructure engineering, currently operating at staff level or above in an individual contributor role.
• Hands-on experience operating production Kubernetes at scale across AWS and GCP.
• Direct, day-to-day experience with inference infrastructure.
• Hands-on experience with NVIDIA GPU systems: drivers, CUDA, NCCL, and node health.
• Hands-on experience with an inference serving framework such as vLLM, SGLang, TensorRT-LLM, or Dynamo.
• Experience with HPC networking, including InfiniBand, RDMA/EFA, or MPI.
• Experience with observability and reliability for latency-sensitive services, including metrics, alerting, and automated failover.
• A track record of building highly available, multi-cloud infrastructure.
• Experience deploying more than one accelerator family, such as Trainium, TPU, or a mixed fleet.
• Previous experience at Meta.

Aufgaben

• Design, operate, and scale Kubernetes across AWS, GCP, and GPU cloud providers, supporting global inference demand at 1,000+ tokens per second.
• Own the production serving layer for LLM inference.
• Debug NVIDIA GPU systems end to end, including drivers, CUDA, NCCL, node health, and the network fabric.
• Work with the model team so performance work holds up in production, on real hardware and real traffic.
• Extend the deployment tooling so a model can be shipped the same way onto NVIDIA, Trainium, and TPU hosts, including mixed fleets.
• Build observability for time to first token, inter-token latency, throughput, and availability, measured per model, per chip, and per region.
• Own reliability engineering: alerting, automated failover, self-healing infrastructure, and traffic routing across models, chips, and regions.

Kenntnisse

Kubernetes at scale

AWS

GCP

NVIDIA GPU expertise

Inference infrastructure

Observability & reliability

Multi-cloud infra

Production systems

Tools

vLLM

TensorRT-LLM

Dynamo

NCCL

Jobbeschreibung

Staff Infrastructure Engineer, Managed Inference

Compensation: $350,000-$450,000

My client is hiring a staff infrastructure engineer to own the production serving layer for managed inference. The role is an individual contributor seat, at staff scope.

What you will do

• Design, operate, and scale Kubernetes across AWS, GCP, and GPU cloud providers, supporting global inference demand at 1,000+ tokens per second.
• Own the production serving layer for LLM inference.
• Debug NVIDIA GPU systems end to end, including drivers, CUDA, NCCL, node health, and the network fabric.
• Work with the model team so performance work holds up in production, on real hardware and real traffic.
• Extend the deployment tooling so a model can be shipped the same way onto NVIDIA, Trainium, and TPU hosts, including mixed fleets.
• Build observability for time to first token, inter-token latency, throughput, and availability, measured per model, per chip, and per region.
• Own reliability engineering: alerting, automated failover, self-healing infrastructure, and traffic routing across models, chips, and regions.

What you bring

• Five or more years of infrastructure engineering, currently operating at staff level or above in an individual contributor role.
• Hands-on experience operating production Kubernetes at scale across AWS and GCP.
• Direct, day-to-day experience with inference infrastructure.
• Hands-on experience with NVIDIA GPU systems: drivers, CUDA, NCCL, and node health.
• Hands-on experience with an inference serving framework such as vLLM, SGLang, TensorRT-LLM, or Dynamo.
• Experience with HPC networking, including InfiniBand, RDMA/EFA, or MPI.
• Experience with observability and reliability for latency-sensitive services, including metrics, alerting, and automated failover.
• A track record of building highly available, multi-cloud infrastructure.
• Helpful, not required
• Experience on a production inference platform, for example Fireworks, Baseten, Nebius, or Anyscale.
• Experience deploying more than one accelerator family, such as Trainium, TPU, or a mixed fleet.
• Previous experience at Meta.

Location and pay

This role is on-site in San Francisco. The range is $350,000-$450,000.