BeaconKV Compresses KV Cache for Long-Reasoning Models with Beacon Queries

Introduction

Extended chain-of-thought reasoning improves the ability of large reasoning models to solve difficult tasks, but it also creates a systems problem. During autoregressive decoding, the Key-Value (KV) cache grows with the sequence. A long reasoning trace can therefore consume a large share of GPU memory, limiting context length and reducing the number of requests that can be served concurrently.

Many existing cache-compression techniques estimate which past tokens will matter by looking at recent queries. This works when attention patterns are mostly local, but long-horizon reasoning can behave differently. A model may generate several intermediate steps and later return to an early plan, condition, or partial conclusion. A policy focused only on the latest queries may remove precisely the information needed for that later revisit.

BeaconKV is designed around this observation. It is a training-free compression method that stores a small set of representative historical queries, called beacon queries, instead of retaining the full query history.

Key ideas

• Reasoning can revisit distant context. The paper calls decoding steps that re-attend to remote earlier content Thought Revisiting Tokens, or TRTs. These steps reflect backtracking, verification, and continued use of an initial problem-solving plan.
• Recent queries are incomplete predictors. A cache policy based only on the newest queries may overvalue local context and discard older KV pairs that become relevant later in the reasoning trace.
• Revisit-related queries form compact groups. The authors observe that queries associated with TRTs tend to cluster into a small number of similarity groups in embedding space. A representative from each group can act as a compact signal for a broader historical access pattern.
• Global and local signals are combined. BeaconKV uses beacon queries alongside recent queries. The recent component captures current attention needs, while the beacon component helps estimate whether an older KV pair could be revisited in the future.
• No model retraining is required. The method changes cache management rather than model parameters. According to the supplied results, experiments on four open-source large reasoning models and diverse reasoning benchmarks show up to 5.8× memory reduction and more than 4.3× higher throughput, while keeping accuracy close to that of a full cache.

Why it matters

The broader contribution of BeaconKV is a different view of cache compression. For short generation, recency can be a useful approximation of importance. Long reasoning is less predictable: the model is not merely extending a sentence, but maintaining and revising a sequence of intermediate decisions. A distant token can remain valuable because it represents a plan or constraint that has not yet been fully resolved.

BeaconKV addresses this issue without preserving every historical query. Instead, it compresses the query-side evidence used to make retention decisions. That makes the approach attractive for inference systems seeking longer reasoning traces, lower memory pressure, or higher serving concurrency without changing model weights.

The reported gains are promising, but the supplied material does not include the full per-model, per-benchmark breakdown or the exact compression configurations. Those details will be important for understanding how stable the trade-off is across architectures, tasks, and hardware. The central idea nevertheless offers a useful direction: effective KV compression for reasoning models may need both recency and a compact memory of recurring global attention patterns.

Source: Hugging Face Daily Papers