Infrastructure Engineer – Managed Inference

Morgan Pinnacle Group; San Francisco (CA); On-site; USD 350,000 - 450,000; Full time

Job summary

Morgan Pinnacle Group is seeking an Infrastructure Engineer to lead production LLM inference infrastructure onsite in San Francisco. The role emphasizes building and operating scalable Kubernetes clusters across AWS, GCP, and GPU cloud providers to support high-throughput inference workloads.

You will own the production inference serving layer, debug GPU driver stacks, and partner with model engineers to ensure performance gains translate to production.

Qualifications

• 5+ years of infrastructure engineering experience.
• Direct, day-to-day experience with production LLM inference infrastructure.
• Extensive experience operating production Kubernetes at scale across multiple cloud providers (AWS, GCP).
• Hands-on expertise debugging NVIDIA GPU and accelerator systems (drivers, CUDA, NCCL, node health, networking).
• Experience with at least one inference serving framework (e.g., vLLM, SGLang, TensorRT-LLM, NVIDIA Dynamo).
• Experience with high-performance networking (InfiniBand, RDMA/EFA, MPI).
• Strong observability and reliability engineering background for latency-sensitive services.
• Ability to diagnose and resolve complex production infrastructure issues.
• Strong communication and collaboration with customers and technical stakeholders.
• Ability to work onsite in San Francisco five days per week.

Responsibilities

• Design, operate, and scale highly available Kubernetes clusters across AWS, GCP, and GPU cloud providers for inference workloads >1,000 tokens/sec.
• Own the production inference serving layer; debug NVIDIA drivers, CUDA, NCCL, node health, and network fabric.
• Collaborate with model optimization engineers to ensure performance translates to production.
• Extend deployment tooling for consistent model deployment across NVIDIA GPUs, AWS Trainium, and Google TPUs.
• Build and maintain observability systems tracking tft, inter-token latency, throughput, and availability.
• Lead reliability engineering with monitoring, alerting, auto-failover, self-healing infra, and traffic routing.
• Troubleshoot production infrastructure issues directly with customers and technical teams.

Skills

5+ years experience

LLM inference infra

Observability & reliability

Communication & problem-solving

Onsite in SF

Tools

Kubernetes

AWS

GCP

NVIDIA GPUs

NCCL

CUDA

vLLM

SGLang

TensorRT-LLM

NVIDIA Dynamo

InfiniBand / RDMA

MPI

Job description

Infrastructure Engineer – Managed Inference

Location: San Francisco, CA (Onsite, 5 days/week)
Employment Type: Full-Time
Compensation: 350,000–450,000 Base Salary + Competitive Equity

About the Opportunity

We are partnering with an innovative AI company building high-performance infrastructure for large-scale generative AI applications. The team develops and operates advanced inference systems designed to deliver exceptionally fast, reliable model serving across multiple cloud providers and hardware architectures.

We are seeking a Staff-level or higher Infrastructure Engineer with deep, hands-on experience operating production LLM inference infrastructure. This role focuses on scaling highly available Kubernetes environments, optimizing inference serving systems, and ensuring reliable performance across GPU and accelerator platforms.

The ideal candidate is a highly technical individual contributor who takes direct ownership of production infrastructure, troubleshoots complex systems, and works closely with engineering teams and customers to solve challenging infrastructure problems.

What You'll Do

• Design, operate, and scale highly available Kubernetes clusters across AWS, GCP, and specialized GPU cloud providers to support inference workloads exceeding 1,000 tokens per second.




• Own the production inference serving layer, including hands-on debugging of NVIDIA drivers, CUDA, NCCL, node health, and network fabric.




• Collaborate with model optimization engineers to ensure performance improvements translate reliably to production environments.




• Extend deployment tooling to support consistent model deployment across NVIDIA GPUs, AWS Trainium, and Google TPUs.




• Build and maintain observability systems tracking time-to-first-token, inter-token latency, throughput, and availability across models, hardware, and regions.




• Lead reliability engineering efforts, including monitoring, alerting, automated failover, self-healing infrastructure, and intelligent traffic routing.




• Troubleshoot production infrastructure issues directly, working with technical teams and customers to identify and resolve complex challenges.

Required Qualifications

• 5+ years of infrastructure engineering experience, currently operating at Staff Engineer level or above in a hands-on individual contributor role.




• Direct, day-to-day experience with production LLM inference infrastructure, including building, deploying, operating, or troubleshooting inference serving systems.




• Extensive experience operating production Kubernetes at scale across multiple cloud providers, particularly AWS and GCP.




• Hands-on expertise debugging NVIDIA GPU and accelerator systems, including drivers, CUDA, NCCL, node health, and network-related issues.




• Practical experience with at least one inference serving framework, such as vLLM, SGLang, TensorRT-LLM, or NVIDIA Dynamo.




• Experience with high-performance networking technologies such as InfiniBand, RDMA/EFA, or MPI.




• Strong background in observability and reliability engineering for latency-sensitive production services, including metrics, alerting, and automated failover.




• Ability to independently diagnose and resolve complex production infrastructure issues.




• Strong communication and problem-solving skills, including the ability to collaborate directly with customers and technical stakeholders.




• Ability to work onsite in San Francisco five days per week.

Preferred Qualifications

• Experience building or operating production-scale inference platforms at AI infrastructure companies, model-serving providers, or AI research organizations.




• Experience deploying inference workloads across multiple accelerator architectures, including NVIDIA GPUs, AWS Trainium, and Google TPUs.




• Contributions to open-source inference frameworks such as vLLM or SGLang.




• Experience supporting high-throughput, low-latency AI applications in production environments.

Technical Environment

Kubernetes, AWS, GCP, NVIDIA GPUs, CUDA, NCCL, InfiniBand, RDMA/EFA, MPI, vLLM, SGLang, TensorRT-LLM, NVIDIA Dynamo, AWS Trainium, Google TPUs, PyTorch, JAX, Docker, Terraform, Prometheus, and Grafana.

Compensation & Location

• Base Salary: 350,000–450,000, with flexibility above this range for exceptionally qualified Staff+ candidates.




• Equity: Competitive equity package.




• Location: San Francisco, California.




• Work Arrangement: Full-time, onsite five days per week.




• Relocation: May be considered for highly qualified candidates.




• Work Authorization: Open to eligible visa transfers, including H-1B and OPT.

Important Considerations

This position requires substantial, direct experience with production LLM inference infrastructure. Candidates whose backgrounds are primarily in general-purpose infrastructure, data engineering, recommendation systems, or inference-adjacent technologies without hands-on inference serving experience are unlikely to meet the requirements.

The team is specifically seeking an experienced Staff-level or higher individual contributor who is already operating at this level and can take immediate ownership of complex production systems.