Staff ML Infra Engineer – LLM Training & Serving Kubernetes

Belvedir (Fractal Machine Research, Inc.); San Francisco (CA); On-site; USD 180,000 - 240,000; Full time

Job summary

Belvedir is seeking a Member of Technical Staff, Infrastructure in San Francisco to build and operate the training and serving stack for custom models on Kubernetes, aiming for scalable distributed training, efficient batching, caching, routing, and autoscaling for latency-sensitive deployments.

You will drive down costs per token with quantization, speculative decoding, kernel work, and smarter scheduling while profiling end-to-end bottlenecks across GPU, CPU, network, or model components to

Qualifications

• Experience training or serving LLMs at scale with PyTorch, vLLM, SGLang, or a similar system.
• Strong systems fundamentals: concurrency, memory, networking, and performance measurement.
• Working knowledge of GPU programming (CUDA or Triton) and transformer training/inference internals.
• Fluency in Python plus a systems language (C++, Rust, or Go) and hands-on GPU workloads on Kubernetes.
• Comfort owning production systems and being on the hook when they break.
• Contributions to an open-source inference engine or kernel library.
• Experience with multi-tenant serving, LoRA serving, or model multiplexing.
• Experience at an early-stage startup or as a founder.

Responsibilities

• Build and operate the training and serving stack for custom models on Kubernetes: distributed training, batching, caching, routing, and autoscaling.
• Drive down cost per token with quantization, speculative decoding, kernel work, and smarter scheduling.
• Profile end to end and fix the bottleneck, wherever it is: GPU, CPU, network, or the model itself.
• Set and hit latency and reliability targets for customer deployments.
• Partner with training on model architectures and formats that are cheap to serve.

Skills

LLM infra training/serving

Systems fundamentals

GPU programming (CUDA/Triton)

Python & systems languages

GPU workloads on Kubernetes

Production systems ownership

Open-source contributions

Multi-tenant/LoRA serving

Startup experience

Job description

Belvedir is seeking a Member of Technical Staff, Infrastructure in San Francisco to build and operate the training and serving stack for custom models on Kubernetes, aiming for scalable distributed training, efficient batching, caching, routing, and autoscaling for latency-sensitive deployments.

You will drive down costs per token with quantization, speculative decoding, kernel work, and smarter scheduling while profiling end-to-end bottlenecks across GPU, CPU, network, or model components to