Member of Technical Staff - Inference

Prime Intellect; San Francisco, Northern (CA, KY); Hybrid; USD 150,000 - 300,000; Full time

Benefits offered by this job

Remote option

Visa sponsorship

Relocation support

Team off-sites

Conference attendance

Job summary

Prime Intellect is building an open frontier AI platform with a focus on scalable LLM serving and RL integration. This hybrid role spans cloud LLM serving, inference optimization, and RL systems.

You will advance our ability to evaluate and serve models trained with our RL Lab at scale, building multi-tenant serving, scheduling, and robust failover across regions. Requires 3+ years on ML/LLM infra, experience with vLLM or TensorRT-LLM, and strong Python/PyTorch skills.

Qualifications

• 3+ years building and running large-scale ML/LLM services with clear latency/availability SLOs.
• Hands-on experience with at least one of vLLM, SGLang, TensorRT-LLM.
• Familiarity with distributed and disaggregated serving infra such as NVIDIA Dynamo.
• Deep understanding of prefill vs. decode, KV-cache, batching, and speculative decoding.

Responsibilities

• Build multi-tenant LLM serving platforms across cloud GPU fleets.
• Design placement and scheduling for heterogeneous accelerators.
• Implement cross-region failover and traffic shifting for resilience and cost control.
• Develop autoscaling, routing, and load balancing to meet SLOs.
• Integrate and optimize LLM inference frameworks (vLLM, SGLang, TensorRT-LLM).
• Collaborate on CI/CD, observability, and incident response for ML infra.

Skills

Python

PyTorch

Kubernetes

CUDA

NCCL

Tools

vLLM

SGLang

TensorRT-LLM

NVIDIA Dynamo

Docker

Job description

Prime Intellect is building the open superintelligence stack: the infrastructure frontier AI labs build internally, made available to every ambitious AI team.

Our platform, Lab, unifies compute, environments, evaluations, secure sandboxes, high-performance training, and deployment into one full-stack system for post-training at frontier scale - from SFT and RL to tool use, agent workflows, and continuously improving production models. We are building open frontier AI: open-source models trained end to end for long-horizon tasks like autonomous research, and the full-stack platform our own research team uses to build them. The next generation of AI companies, enterprises, and research teams do not just need more GPUs. They need the ability to turn their own workflows, tools, data, and feedback loops into superintelligence they own.

Role Impact

This is a hybrid position spanning cloud LLM serving, LLM inference optimization and RL systems. You will be working on advancing our ability to evaluate and serve models trained with our RL Lab at scale. The two key areas are:

Building the infrastructure to serve LLMs efficiently at scale.

Optimization and integration of inference systems into our RL training stack.

Core Technical Responsibilities

LLM Serving

Multi‑tenant LLM Serving: Build a multi-tenant LLM serving platform that operates across our cloud GPU fleets.

GPU‑Aware Scheduling: Design placement and scheduling algorithms for heterogeneous accelerators.

Resilience & Failover: Implement multi‑region/zone failover and traffic shifting for resilience and cost control.

Autoscaling & Routing: Build autoscaling, routing, and load balancing to meet throughput/latency SLOs.

Model Distribution: Optimize model distribution and cold‑start times across clusters.

Inference Optimization & Performance

Framework Development: Integrate and contribute to LLM inference frameworks such as vLLM, SGLang, TensorRT‑LLM.

Parallelism and Configuration Tuning: Optimize configurations for tensor/pipeline/expert parallelism, prefix caching, memory management and other axes for maximum performance.

End‑to‑End Performance: Profile kernels, memory bandwidth and transport; apply techniques such as quantization and speculative decoding.

Perf Suites: Develop reproducible performance suites (latency, throughput, context length, batch size, precision).

RL Integration: Embed and optimize distributed inference within our RL stack.

Platform & Tooling

CI/CD: Establish CI/CD with artifact promotion, performance gates, and reproducible builds.

Observability: Build metrics, logs, tracing; structured incident response and SLO management.

Docs & Collaboration: Document architectures, playbooks, and API contracts; mentor and collaborate cross‑functionally.

Technical Requirements

Required Experience

Building ML Systems at Scale: 3+ years building and running large‑scale ML/LLM services with clear latency/availability SLOs.

Inference Backends: Hands‑on with at least one of vLLM, SGLang, TensorRT‑LLM.

Distributed Serving Infra: Familiarity with distributed and disaggregated serving infrastructure such as NVIDIA Dynamo.

Inference Internals: Deep understanding of prefill vs. decode, KV‑cache behavior, batching, sampling, speculative decoding, parallelism strategies.

Full‑Stack Debugging: Comfortable debugging CUDA/NCCL, drivers/kernels, containers, service mesh/networking, and storage, owning incidents end‑to‑end.

Infrastructure Skills

Python: Systems tooling and backend services.

PyTorch: LLM Inference engine development and integration, deployment readiness.

Kubernetes: Running infrastructure at scale with containers on Kubernetes.

GPU & Networking: Architecture, CUDA runtime, NCCL, InfiniBand; GPU‑aware bin‑packing and scheduling across heterogeneous fleets.

Nice to Have

Kernel‑Level Optimization: Familiarity with CUDA/Triton kernel development; Nsight Systems/Compute profiling.

Systems Performance Languages: Rust, C++.

Data & Observability: Kafka/PubSub, Redis, gRPC/Protobuf; Prometheus/Grafana, OpenTelemetry; reliability patterns.

Open Source: Contributions to serving, inference, or RL infrastructure projects.

What We Offer

Cash Compensation Range of $150-300kwith significant equity incentives

Flexible work arrangement (remote or San Francisco office)

Full visa sponsorship and relocation support

Regular team off‑sites and conference attendance

Opportunity to shape decentralized AI and RL at Prime Intellect

You'll join a team of experienced engineers and researchers working on cutting‑edge problems in AI infrastructure. We believe in open development and encourage team members to contribute to the broader AI community through research and open‑source contributions.

We value potential over perfection. If you're passionate about democratizing AI development, we want to talk to you.

Ready to help shape the future of AI?