vLLM’s Bidirectional KV Transfer Cuts Repeated Prefill in Agentic Chats

Introduction

Agentic conversations accumulate context as a model reasons, calls tools, and incorporates the returned results. On every new turn, the client commonly resends system instructions, previous requests, tool outputs, and earlier model responses. In a Prefill-Decode, or P-D, deployment, the prefill worker processes that prompt before the decode worker generates new tokens. If no previous KV state is available to the prefill side, the same history must be processed again.

vLLM’s bidirectional KV transfer is designed to remove part of that duplication. Conventional disaggregated serving primarily moves KV from prefill to decode. The new flow also lets a decode node retain eligible KV blocks after generation and make them available to prefill on the next turn.

Key points

• Reuse works in both directions. The first request follows the conventional path: prefill computes prompt KV and decode reads it. After generation, decode retains relevant prompt and response blocks. On a later turn, prefill can retrieve the reusable prefix and compute only the new suffix.
• The router handles metadata, not tensors. A KV-aware router associates a client-provided conversation_id with decode’s block-location metadata. It receives that information from the response stream and attaches it to the next prefill request. KV tensors move directly between workers through NIXL rather than passing through the router.
• Retention is bounded. Decode normally releases request blocks when generation ends. In bidirectional mode, it keeps them for a configurable lifetime; decoder_kv_blocks_ttl defaults to 480 seconds in the described setup. Prefill first checks its local prefix cache and starts an asynchronous remote read only when the reusable amount reaches kv_recompute_threshold.
• The worker roles remain distinct. Prefill computes KV for new prompt tokens, while decode supplies already cached blocks. The NIXL READ mechanism is shared across P-to-D and D-to-P transfers, but each worker type maintains the metadata logic appropriate to its role.

Performance and implications

The reported multi-turn tests used a 1P1D deployment on Amazon EC2 P5en instances connected with AWS Elastic Fabric Adapter. Bidirectional transfer improved TTFT by up to 1.6x for Qwen3-32B and up to 3.3x for zai-org/GLM-5-FP8. These are workload- and configuration-dependent results, not universal guarantees.

The broader benefit is a change in cache ownership across the serving pipeline. Instead of treating generated KV as disposable once decoding ends, the system keeps useful state available for the next turn. That can reduce repeated prefill computation, improve accelerator utilization, and help long-running assistants maintain more stable first-token latency as their histories grow.

The design also adds operational requirements. A deployment needs a stateful KV-aware router, reliable conversation identifiers, expiration handling, and a fallback path when a remote block is unavailable. TTL, the recompute threshold, local cache behavior, network performance, and session placement will all influence whether remote reuse outweighs its transfer cost.

vLLM says the capability was first built and validated on AWS Trainium clusters with EFA and then contributed as accelerator-agnostic functionality. The results in this post come from an EFA-connected GPU cluster, showing that the same serving pattern can extend beyond the original accelerator environment.

Source: vLLM Blog