vLLM’s Tiered KV Cache Turns Host Memory into a Shared Inference Layer

Long-context requests and multi-turn conversations can produce KV caches that quickly exceed accelerator memory. Once older entries are evicted, a later request traditionally has to recompute them. vLLM’s Tiered KV Cache Offloading introduces a persistence path for those evicted chunks, allowing the serving stack to reload data from a lower tier instead of always repeating the original computation.

A host-centric data path

The key architectural choice is to make CPU memory the hub for every KV movement. During offload, data is copied from the accelerator to host DRAM first. As soon as that local copy finishes, accelerator memory can be released; writes to storage or transfers to peers continue asynchronously from the host copy. On reload, the sequence is reversed: data is promoted into host memory and only then placed back on the accelerator. This creates a just-in-time allocation pattern in which GPU memory is reserved primarily when data is ready to be used.

Host memory is a real LRU/ARC cache rather than a temporary staging area. The scheduler checks it first. On a miss, it queries secondary tiers in their configured order, and the first tier containing the chunk serves it. Separate chunks belonging to one request may therefore be sourced from different tiers while promotion proceeds asynchronously.

A canonical representation for sharing

In tensor-parallel deployments, each accelerator holds a shard of the KV cache. vLLM consolidates those shards into a shared host-memory region and uses a canonical layout: a page represents a block from one layer, with KV heads from all tensor-parallel ranks gathered contiguously. This reduces the number of secondary-tier operations and turns many small transfers into fewer, larger I/O requests.

The fixed host-side representation also separates cache sharing from accelerator-specific details. Nodes using different accelerator types, attention backends, or parallelism settings can exchange the same host chunks without remapping their GPU layouts. That makes the cache easier to share across heterogeneous deployments.

Storage tiers and practical uses

• Filesystem storage saves content-addressed chunks on local or networked filesystems, allowing instances sharing a mount to reuse the same data.
• Object storage uses S3-compatible services as a more economical, network-accessible tier.
• Peer-to-peer transfer combines coordination through ZMQ with host-to-host bulk movement over RDMA via NIXL.

The peer tier is useful for prefill/decode disaggregation and load balancing. A prefill worker can expose completed chunks from host memory, while a decode worker retrieves them remotely. With chunked prefill, computation and movement can overlap earlier. Similarly, an overloaded instance can make data available to another instance with spare capacity. Consolidating accelerator shards before transfer can improve network efficiency compared with independent GPU-level movements.

Why it matters

The broader significance is that KV Cache becomes a tiered, persistent, and shareable infrastructure resource rather than a private accelerator-side working set. The approach can reduce recomputation and expand effective serving capacity, while simplifying secondary-tier implementations because they only need standard CPU-side I/O. Its benefits still depend on host capacity, storage latency, network bandwidth, and cache locality; remote storage avoids recomputation but does not make data movement free. Even so, the design gives vLLM a practical foundation for scaling context state across machines.

vLLM Blog