vLLM’s Bidirectional KV Transfer Cuts Repeated Prefill in Agentic Chats
vLLM has added bidirectional KV transfer to its Prefill-Decode serving design, allowing prefill workers to reuse context retained by decode workers on later turns. The approach reduced TTFT by up to 3.3x in the reported long-context benchmarks.
Read more