vLLM Brings Hybrid HiSparse Offloading to GLM 5.3 Inference

Introduction

The deployment challenge for long-context models is rarely whether one request can complete. The harder problem appears when many agent sessions grow at the same time: every new turn consumes more KV cache, while the GPU block pool remains fixed. Traditional systems generally choose between preemption and offloading. Preemption frees space by discarding a request’s KV cache, but the request must pay the prefill cost again. Conventional offloading preserves the cache in host memory, yet dense attention still requires the full history to return to the GPU before decoding can continue.

vLLM’s GLM 5.3 work introduces Hybrid HiSparse to exploit a property of sparse MLA: the model does not need every historical token for each decode step.

Key points

• Offload only under pressure. KV pages remain resident on the GPU while capacity is available. Older pages are released progressively only when the shared GPU block pool becomes constrained.
• Use Top-K access patterns. The sparse MLA indexer selects a small set of historical tokens. HiSparse keeps most of the selected model KV in host memory and brings the rows requested by the indexer into GPU hot buffers. The indexer KV itself remains outside this mechanism and can use the standard offloading connector.
• Share one allocation pool. Hot pages are ordinary KV-cache blocks leased through vLLM’s Hybrid Memory Allocator, rather than a completely separate memory region. A request can therefore combine resident GPU pages, CPU-only pages, and hot-buffer pages.
• Keep decoding alive. Resident rows are read in place, hot hits are reused, and a miss copies one row from pinned host memory into an LRU slot. The request does not need to wait for its entire history to fit on the GPU again, nor does it need to be re-prefilled after every eviction.
• Compose with the rest of vLLM. Prefix caching, other cache groups, P/D-disaggregation imports, speculative decoding, and ordinary KV offloading can continue to use their existing paths.

How the residency policy works

Hybrid HiSparse tracks three broad states. In full residency, sparse-MLA KV remains on the GPU while completed prefix pages are proactively copied to host memory. In mixed residency, the request tail stays on the GPU, older pages exist only on the CPU, and the rows selected from those pages are placed in hot buffers. In the no-residency case, a request reusing a prefix that is available only on the CPU starts with placeholders and loads rows as the indexer asks for them.

The hot buffers live in the same KV-cache tensor and draw blocks from the same pool as ordinary resident pages. This lets the resolver return a unified set of row identifiers to the allocator and gather them with the same stride. A page freed by one request can become hot-buffer capacity for another. The design also prepares host copies before pressure arrives: once a cacheable prefix page is complete, HiSparse queues a copy while continuing to serve the GPU version. If memory later runs short, that page can surrender its GPU slot without another transfer.

The implementation copies sparse-MLA layers together after the forward pass and orders the transfer on the model’s GPU stream. The decode path does not wait for a CPU-side residency decision, which preserves compatibility with CUDA Graph capture. Since MLA KV is identical across tensor-parallel ranks, the pinned host pool is organized per data-parallel replica and shared by its local ranks, with CUDA events maintaining ordering.

Evaluation and implications

The published evaluation used a single 8×H200 node, GLM 5.3, MTP3, FP8 KV cache, and an OpenHands multi-turn agent workload. The first turn contained a long context, later turns were shorter, and each response used a fixed output length. The comparison used conventional KV offloading with the same overall host-memory budget, while Hybrid HiSparse divided that budget between sparse offloading and the standard path. Measurements covered the interactivity-throughput trade-off and the number of running requests across context lengths.

The reported behavior is important: requests can continue decoding while only part of their history remains GPU-resident, enabling higher concurrency and making the full context target practical on the tested hardware. The exact benefit depends on sparse-access patterns, hot-buffer hit rates, CPU-GPU bandwidth, and workload shape. HiSparse is therefore not a universal replacement for dense KV offloading. Its broader significance is architectural: memory residency can be driven by the tokens the model is likely to read, instead of treating the entire context as equally valuable. vLLM plans to make Hybrid HiSparse more broadly available in a future release.

Source: vLLM Blog