Staff ML Systems Engineer - LLM Serving & RL

Prime Intellect; San Francisco, Northern (CA, KY); Hybrid; USD 150,000 - 300,000; Full time

Benefits offered by this job

Remote option

Visa sponsorship

Relocation support

Team off-sites

Conference attendance

Job summary

Prime Intellect is building an open frontier AI platform with a focus on scalable LLM serving and RL integration. This hybrid role spans cloud LLM serving, inference optimization, and RL systems.

You will advance our ability to evaluate and serve models trained with our RL Lab at scale, building multi-tenant serving, scheduling, and robust failover across regions. Requires 3+ years on ML/LLM infra, experience with vLLM or TensorRT-LLM, and strong Python/PyTorch skills.

Qualifications

• 3+ years building and running large-scale ML/LLM services with clear latency/availability SLOs.
• Hands-on experience with at least one of vLLM, SGLang, TensorRT-LLM.
• Familiarity with distributed and disaggregated serving infra such as NVIDIA Dynamo.
• Deep understanding of prefill vs. decode, KV-cache, batching, and speculative decoding.

Responsibilities

• Build multi-tenant LLM serving platforms across cloud GPU fleets.
• Design placement and scheduling for heterogeneous accelerators.
• Implement cross-region failover and traffic shifting for resilience and cost control.
• Develop autoscaling, routing, and load balancing to meet SLOs.
• Integrate and optimize LLM inference frameworks (vLLM, SGLang, TensorRT-LLM).
• Collaborate on CI/CD, observability, and incident response for ML infra.

Skills

Python

PyTorch

Kubernetes

CUDA

NCCL

Tools

vLLM

SGLang

TensorRT-LLM

NVIDIA Dynamo

Docker

Job description

Prime Intellect is building an open frontier AI platform with a focus on scalable LLM serving and RL integration. This hybrid role spans cloud LLM serving, inference optimization, and RL systems.

You will advance our ability to evaluate and serve models trained with our RL Lab at scale, building multi-tenant serving, scheduling, and robust failover across regions. Requires 3+ years on ML/LLM infra, experience with vLLM or TensorRT-LLM, and strong Python/PyTorch skills.