Senior Principal LLM Bring‑up & Inference Architect

SambaNova, Inc.; San Jose (CA); On-site; USD 208,000 - 312,000; Full time

Benefits offered by this job

Health insurance

HSA

Dental

Vision

Wellbeing benefits

Job summary

SambaNova Systems, based in San Jose, CA, is seeking a senior ML systems engineer to bring up open-weight LLMs on our RDU accelerators. You will adapt models, map across chips and hosts, compile, and validate numerically against references, then deploy production serving with latency and throughput targets.

You will build shared inference features such as continuous batching, prompt caching, and speculative decoding (FP8/FP4), and contribute to MLIR/XLA/Triton based compilers and tooling.

Qualifications

• B.S. in Computer Science, Electrical Engineering, or related field.
• 8+ years of industry experience in ML systems, inference, or compiler engineering.
• Strong Python and PyTorch internals experience, plus some C++.
• Deep transformer-inference knowledge, including KV caching, GQA and sliding window attention, MoE routing, prefill vs decode, batching, and speculative decoding.
• Hands‑on experience with accelerator bring‑up, ML compilers (MLIR, XLA, TVM, Triton, torch‑mlir), or inference engines (vLLM, SGLang, TensorRT‑LLM).
• Rigor in numerical debugging.
• Comfort with Bazel, Slurm, and large monorepos.
• Clear writing and evidence‑backed claims.

Responsibilities

• Bring up new LLM architectures end to end: model adaptation, checkpoint conversion, multi-chip / multi-host mapping, compile, strict numerical validation, production serving with latency / throughput numbers.
• Build inference features once as shared modules: continuous batching over a paged KV pool, prompt caching, chunked and incremental prefill, dynamic tensor shapes, speculative decoding (EAGLE-3, drafters, MTP), FP8 / FP4, tensor and expert parallelism.
• Debug where model, compiler and hardware meet; land each fix in the right shared component.

Skills

Python

PyTorch internals

C++

Transformer inference

ML compilers

Numerical debugging

Bazel

Slurm

Technical writing

GQA attention

Education

B.S. in Computer Science / Electrical Engineering

M.S. or Ph.D. in related field

Tools

MLIR

XLA

TVM

Triton

torch-mlir

TensorRT-LLM

Job description

SambaNova Systems, based in San Jose, CA, is seeking a senior ML systems engineer to bring up open-weight LLMs on our RDU accelerators. You will adapt models, map across chips and hosts, compile, and validate numerically against references, then deploy production serving with latency and throughput targets.

You will build shared inference features such as continuous batching, prompt caching, and speculative decoding (FP8/FP4), and contribute to MLIR/XLA/Triton based compilers and tooling.