Staff Engineer - LLM Serving & Inference at Scale

Primeintellect; San Francisco (CA); Hybrid; USD 150,000 - 300,000; Full time

Benefits offered by this job

Remote or SF office

Visa sponsorship

Relocation support

Professional development budget

Team off-sites

Equity incentives

Job summary

Prime Intellect in San Francisco is seeking an experienced ML systems engineer to scale LLM serving and RL components across cloud GPU fleets.

You will build multi-tenant serving platforms, optimize inference and integrate frameworks like vLLM, SGLang, and TensorRT-LLM, while implementing robust CI/CD and observability.

This hybrid role offers remote or San Francisco office options, visa sponsorship, relocation support, and meaningful equity to shape frontier AI infrastructure.

Qualifications

• 3+ years building and running large-scale ML/LLM services with clear latency/availability SLOs.
• Hands-on with at least one of vLLM, SGLang, TensorRT-LLM.
• Familiarity with distributed and disaggregated serving infrastructure such as NVIDIA Dynamo.
• Deep understanding of prefill vs. decode, KV-cache, batching, sampling, speculative decoding, parallelism strategies.
• Full-Stack debugging across CUDA/NCCL, drivers/kernels, containers, service mesh and storage.

Responsibilities

• Build a multi-tenant LLM serving platform across cloud GPU fleets.
• Design placement and scheduling for heterogeneous accelerators.
• Implement multi-region/zone failover and traffic shifting for resilience and cost control.
• Develop autoscaling, routing, and load balancing for throughput and latency SLOs.
• Optimize model distribution and cold-start times across clusters.
• Integrate and contribute to LLM inference frameworks like vLLM, SGLang, TensorRT-LLM.
• Tune configurations for tensor/pipeline/expert parallelism and memory management.
• Profile kernels and memory bandwidth; apply quantization and speculative decoding.
• Embed and optimize distributed inference within our RL stack.
• Establish CI/CD with reproducible builds and performance gates.
• Build observability via metrics, logs, tracing; incident response and SLO management.

Skills

ML systems at scale

Inference backends

Distributed serving

Inference internals

Full-stack debugging

Tools

vLLM

SGLang

TensorRT-LLM

NVIDIA Dynamo

Job description

Prime Intellect in San Francisco is seeking an experienced ML systems engineer to scale LLM serving and RL components across cloud GPU fleets.

You will build multi-tenant serving platforms, optimize inference and integrate frameworks like vLLM, SGLang, and TensorRT-LLM, while implementing robust CI/CD and observability.

This hybrid role offers remote or San Francisco office options, visa sponsorship, relocation support, and meaningful equity to shape frontier AI infrastructure.