Member of Technical Staff, Inference Systems

Wintermeyer Ventures

Palo Alto (CA)

On-site

USD 230,000 - 350,000

Full time

25 hours ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Wintermeyer Ventures is hiring for an experienced backend/distributed systems engineer to help build a high-performance inference platform for large language models. You will shape runtime architecture, optimize KV caching, and scale serving across GPUs and nodes.

The role requires hands-on experience with LLM inference systems, transformer internals, and production engines like vLLM, SGLang, or TensorRT-LLM. Strong systems programming skills in Rust, C++, Go, or Python are essential.

Qualifications

  • Two to ten years of backend or distributed systems engineering experience.
  • Hands-on experience building, operating, or optimizing LLM inference or serving systems, beyond using hosted APIs.
  • Strong understanding of transformer inference, including attention, KV cache, batching, scheduling, and performance bottlenecks.
  • Experience with production inference engines such as vLLM, SGLang, or TensorRT-LLM, and performance-critical systems where latency, throughput, and cost matter.
  • Strong systems programming skills in Rust, C++, Go, or systems-level Python and PyTorch, with the ability to become productive in Rust quickly.

Responsibilities

  • Build a Rust inference runtime, owning batching, scheduling, request routing, and the serving stack.
  • Design KV cache management, prefix caching, and other optimizations to reduce latency and cost per token.
  • Scale inference across multiple GPUs and nodes.
  • Profile and benchmark the inference pipeline, then deliver performance improvements.
  • Collaborate with the team on core architectural decisions.

Skills

Backend engineering
Distributed systems
Rust
C++
Go
Python
PyTorch
Performance tuning
LLM inference
System design

Tools

vLLM
SGLang
TensorRT-LLM
Triton
CUDA

Job description

About the Role

Join a small AI systems team building a high-performance inference platform from the ground up. You will shape the runtime architecture and improve how large language models are served across GPUs and nodes.

What You'll Do
  • Build a Rust inference runtime, owning batching, scheduling, request routing, and the serving stack.
  • Design KV cache management, prefix caching, and other optimizations to reduce latency and cost per token.
  • Scale inference across multiple GPUs and nodes.
  • Profile and benchmark the inference pipeline, then deliver performance improvements.
  • Collaborate with the team on core architectural decisions.
What We're Looking For
  • Two to ten years of backend or distributed systems engineering experience.
  • Hands-on experience building, operating, or optimizing LLM inference or serving systems, beyond using hosted APIs.
  • Strong understanding of transformer inference, including attention, KV cache, batching, scheduling, and performance bottlenecks.
  • Experience with production inference engines such as vLLM, SGLang, or TensorRT-LLM, and performance-critical systems where latency, throughput, and cost matter.
  • Strong systems programming skills in Rust, C++, Go, or systems-level Python and PyTorch, with the ability to become productive in Rust quickly.
  • Experience with multi-GPU or multi-node serving, CUDA or Triton, or techniques such as prefix caching, speculative decoding, and prefill/decode disaggregation is valuable.
  • Strong computer science and systems fundamentals.
Compensation & Benefits

Base salary: $230,000 to $350,000 annually. Visa sponsorship is available.

Location

On-site in Palo Alto, California, United States.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Member of Technical Staff, Inference
Member of Technical Staff, Inference

OpenRelay, Inc. • San Francisco (CA)

Hybrid
USD 180,000 - 240,000
Member of Technical Staff, Inference
Member of Technical Staff, Inference

Inferact • San Francisco (CA)

Hybrid
USD 200,000 - 400,000
Health, dental, and vision benefits
401(k) company match
Visa sponsorship on case-by-case basis
Senior Software Engineer - Model Performance
Senior Software Engineer - Model Performance

inference.net • San Francisco (CA)

Hybrid
USD 220,000 - 320,000
Equity in a high-growth startup
Comprehensive benefits
Member of Technical Staff — Inference Infrastructure
Member of Technical Staff — Inference Infrastructure

Kindredventures • San Francisco (CA)

On-site
USD 180,000 - 240,000
Tech Lead - AI Inference
Tech Lead - AI Inference

wekatest • United States

Remote
USD 180,000 - 240,000
Medical insurance
Dental insurance
Vision insurance
+5
Senior Software Engineer - Model Performance
Senior Software Engineer - Model Performance

Inference • San Francisco (CA)

Hybrid
USD 220,000 - 320,000
Competitive compensation
Equity in a high-growth startup
Comprehensive benefits
Member of Technical Staff, Performance and Scale
Member of Technical Staff, Performance and Scale

Inferact • San Francisco (CA)

Hybrid
USD 200,000 - 400,000
Generous health benefits
Dental and vision benefits
401(k) company match
Member of Technical Staff (AI Inference Engineer)
Member of Technical Staff (AI Inference Engineer)

Kindredventures • Palo Alto (CA)

Hybrid
USD 190,000 - 250,000
Comprehensive health insurance
Dental insurance
Vision insurance
+1
Senior Infrastructure Engineer
Senior Infrastructure Engineer

Xpertise Recruitment • San Francisco (CA)

On-site
USD 350,000 - 450,000
Software Engineer, Inference Systems
Software Engineer, Inference Systems

River AI • Palo Alto (CA)

On-site
USD 200,000 - 420,000
Health insurance
Dental insurance
Vision insurance
+3