LLM Inference KV Cache Engineer (Distributed)

ByteDance; Seattle (WA); On-site; USD 154,000 - 301,000; Full time

Job summary

ByteDance’s Infrastructure System Lab in Seattle is seeking a Research Engineer/Scientist to design a high-performance KV caching layer for LLM inference across GPUs and nodes. The role targets low latency, high throughput, and cost efficiency by reusing attention keys/values and embeddings in transformer models.

You will collaborate with inference and serving teams to integrate the cache with token streaming pipelines, memory-aware sharding, and multi-tenant consistency, with opportunities to

Qualifications

• PhD in Computer Science, Applied Mathematics, Electrical Engineering or related field.
• Strong understanding of transformer internals and KV caching impact on decoding.
• Experience with low-latency, distributed systems and GPU-aware networking.

Responsibilities

• Design and implement a distributed KV cache to store intermediate states for transformer LLMs across GPUs/nodes.
• Optimize latency, throughput, and eviction policies for long-context caching and reused embeddings.
• Collaborate with inference/serving teams to integrate cache with token streaming, batched decoding, and model parallelism.
• Develop cache consistency protocols for multi-tenant environments.
• Implement memory-aware sharding, eviction (LRU/TTL), and replication strategies across GPUs or distributed backends.
• Monitor performance and iterate caching algorithms to reduce compute costs and latency.

Skills

Transformer internals

KV caching

Distributed systems

C/C++/Rust/Go

CUDA programming

Education

PhD in CS / math / EE

Tools

CUDA

TensorRT

Triton Inference Server

NCCL

Job description

ByteDance’s Infrastructure System Lab in Seattle is seeking a Research Engineer/Scientist to design a high-performance KV caching layer for LLM inference across GPUs and nodes. The role targets low latency, high throughput, and cost efficiency by reusing attention keys/values and embeddings in transformer models.

You will collaborate with inference and serving teams to integrate the cache with token streaming pipelines, memory-aware sharding, and multi-tenant consistency, with opportunities to