Member of Technical Staff, Inference Infrastructure

Kindredventures

Palo Alto (CA)

On-site

USD 190,000 - 240,000

Full time

5 days ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Kindredventures is seeking a senior engineer to design and own the systems that serve our models in production. You will build a fast, reliable inference platform for many specialized models, ensure secure deployments, and collaborate with the RL team to accelerate training.

You will optimize latency and throughput with batching, caching, and scheduling while mentoring other engineers and shaping our technical direction.

Qualifications

  • Experience building or operating ML inference systems in production.
  • Deep experience with LLM serving including KV caching, continuous batching, request scheduling, and parallelism strategies.
  • Hands-on experience with inference engines such as vLLM, SGLang, or TensorRT-LLM, ideally including working on their internals.
  • Strong Python skills; C++, Rust, or Go is a plus.
  • Experience with Kubernetes, autoscaling, load balancing, and running GPU fleets across cloud providers.
  • Strong understanding of transformer architectures and what drives inference performance.
  • A track record of owning critical production systems, including incident response.
  • Comfortable with ambiguity and able to own large problems end to end.

Responsibilities

  • Design, build, and operate our production inference platform for serving many specialized models at scale.
  • Improve latency, throughput, and GPU utilization using batching, caching, scheduling, quantization, and speculative decoding.
  • Build systems to serve many fine-tuned model variants efficiently on shared hardware.
  • Build safe, fast rollout systems for new model versions, including canary releases, A/B tests, and quick rollbacks.
  • Support deployments in customer cloud environments and VPCs, with strong security and isolation.
  • Work with the RL infrastructure team to make inference fast and efficient for rollout generation during training.
  • Capture production traces and feedback to inform training.
  • Build monitoring, alerting, and on-call practices to keep the platform reliable.
  • Evaluate and adopt new models, hardware, and inference engines as they appear.
  • Set engineering standards and mentor other engineers as the team grows.

Skills

ML Inference Systems
LLM Serving
Python
Ambiguity Tolerance
Incident Response

Tools

vLLM
SGLang
TensorRT-LLM
Kubernetes
Autoscaling
GPU Scheduling
CUDA
Triton

Job description

About Us

We are a small team of researchers and engineers focused on pushing the frontier of what agents are capable of. We believe the next leap in agent performance will come not from bigger general models, but from specialization and continual learning: agents that get better at their specific job the more they do it. We build the systems that make this possible, turning real-world usage into lasting improvement.

Summary

We are looking for a senior engineer to design and own the systems that serve our models in production. Our customers run many specialized models that are updated often as they learn from real-world use. You will build inference infrastructure that is fast, reliable, and cost-efficient, and that makes it easy to deploy new model versions safely. You will also work closely with the RL team, since fast inference is at the heart of RL training. You will have a large influence on our technical direction.

Responsibilities
  • Design, build, and operate our production inference platform for serving many specialized models at scale.

  • Improve latency, throughput, and GPU utilization using techniques such as batching, caching, scheduling, quantization, and speculative decoding.

  • Build systems to serve many fine-tuned model variants efficiently on shared hardware.

  • Build safe, fast rollout systems for new model versions, including canary releases, A/B tests, and quick rollbacks.

  • Support deployments in customer cloud environments and VPCs, with strong security and isolation.

  • Work with the RL infrastructure team to make inference fast and efficient for rollout generation during training.

  • Capture production traces and feedback in a way that can feed back into training.

  • Build monitoring, alerting, and on-call practices that keep the platform reliable.

  • Evaluate and adopt new models, hardware, and inference engines as they appear.

  • Set engineering standards and mentor other engineers as the team grows.

Qualifications
  • Experience building or operating ML inference systems in production.

  • Deep experience with LLM serving, including KV caching, continuous batching, request scheduling, and parallelism strategies.

  • Hands‑on experience with inference engines such as vLLM, SGLang, or TensorRT-LLM, ideally including working on their internals.

  • Strong Python skills. C++, Rust, or Go is a plus.

  • Experience with Kubernetes, autoscaling, load balancing, and running GPU fleets across cloud providers.

  • Strong understanding of transformer architectures and what drives inference performance.

  • A track record of owning critical production systems, including incident response.

  • Comfortable with ambiguity and able to own large problems end to end.

Nice to Have
  • Experience building inference infrastructure at a frontier lab or AI infrastructure company.

  • Contributions to open-source inference engines.

  • Experience with GPU kernel programming (CUDA, Triton) and profiling.

  • Experience with multi-LoRA serving, disaggregated prefill and decode, or Mixture of Experts models.

  • Experience integrating inference with RL training systems.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Member of Technical Staff, Infrastructure Lead (RL and Inference)
Member of Technical Staff, Infrastructure Lead (RL and Inference)

Kindredventures • Palo Alto (CA)

On-site
USD 230,000 - 360,000
Member of Technical Staff, RL Infrastructure
Member of Technical Staff, RL Infrastructure

Kindredventures • Palo Alto (CA)

On-site
USD 150,000 - 230,000
Member of Technical Staff, Inference
Member of Technical Staff, Inference

OpenRelay, Inc. • San Francisco (CA)

Hybrid
USD 180,000 - 240,000
INFERENCE ENGINEER
INFERENCE ENGINEER

Makermaker • San Francisco (CA)

On-site
USD 120,000 - 160,000
Member of Technical Staff, Inference Systems
Member of Technical Staff, Inference Systems

Wintermeyer Ventures • Palo Alto (CA)

On-site
USD 230,000 - 350,000
Member of Technical Staff — Inference Infrastructure
Member of Technical Staff — Inference Infrastructure

Kindredventures • San Francisco (CA)

On-site
USD 180,000 - 240,000
AI Infrastructure Engineer
AI Infrastructure Engineer

Pokee AI • United States

Remote
USD 140,000 - 220,000
Inference Engineer - Member of Technical Staff
Inference Engineer - Member of Technical Staff

Designworks Talent • Bellevue (WA)

Hybrid
USD 150,000 - 230,000
Medical, dental, and vision insurance
401(k) with company match
Paid holidays
Member of Technical Staff, Applied AI Engineer
Member of Technical Staff, Applied AI Engineer

Kindredventures • Palo Alto (CA)

On-site
USD 190,000 - 250,000
Tech Lead - AI Inference
Tech Lead - AI Inference

wekatest • United States

Remote
USD 180,000 - 240,000
Medical insurance
Dental insurance
Vision insurance
+5