vLLM’s DCP Tackles the KV Cache Bottleneck in Long-Context Inference

Introduction

Long-context inference has moved from a benchmark feature to an operational requirement for agentic systems. Assistants that inspect repositories, maintain extended chat histories, or replay multi-turn tool traces can easily face inputs ranging from tens of thousands of tokens to much more. vLLM’s blog argues that in this regime, the key constraint is often the decode-time KV cache: it grows with context length and can crowd out the memory needed to serve additional requests.

Key points

• Tensor parallelism has a hard limit for KV sharding. Standard TP partitions KV cache by attention head. That works only while there are enough KV heads to distribute. GQA models use relatively few KV heads, so sharding quickly bottoms out. MLA models make the situation even tighter because Key/Value states are compressed into a shared low-rank latent representation, leaving little or nothing to split by head.
• DCP shards by sequence position. Decode Context Parallelism assigns different ranges of the same sequence’s KV cache to different GPUs. A 200K-token request, for example, can have token-position ranges stored across multiple devices. Each GPU then keeps only a fraction of the cache, freeing memory for larger batches and more concurrent requests.
• The decode communication pattern is manageable. DCP gathers the query, performs attention against each GPU’s local KV slice, and then merges partial outputs using an all-gather plus reduce-scatter pattern. Because decode operates on a single new token at a time, gathering the query is relatively cheap. For MLA, vLLM also exposes an optional query-projection replication path to skip that query all-gather.
• The measured gain appears in high-concurrency long-context serving. In the blog’s experiment on a single 8×B200 node serving Kimi K2.6 in NVFP4, baseline TP reached full KV usage at concurrency 64 and plateaued around 1,863 tok/s/GPU. DCP continued scaling to concurrency 512, reached 6,091 tok/s/GPU, and still reported 82% KV usage. The blog summarizes the result as about 3× higher throughput for the long-context agentic workload.

Why it matters

DCP is less about accelerating one isolated long request and more about keeping a serving system usable when many long-context sessions arrive at once. By reducing per-GPU KV cache pressure, vLLM can admit more requests and operate at larger batch sizes without immediately hitting the memory wall. That is directly relevant to agent products, where interactivity and cost per token both depend on how many sessions can share the same GPU fleet.

The design also reflects a broader shift in inference optimization. As GQA and MLA reduce KV representation size, traditional head-based partitioning becomes less effective. Sequence-dimension sharding gives serving systems a different scaling axis, one that matches the reality of 64K-to-1M-token traces. For long-context agents, cache layout and GPU-to-GPU communication are becoming as important as model weights or kernel speed.

Source: vLLM Blog