KV Cache Eviction Has a Structural Bias Problem, New arXiv Paper Finds

Lead

KV cache compression is becoming a central technique for making long-context large language models practical under memory constraints. When the context grows, the inference system must decide which historical Key/Value states to keep. Many methods, including H2O and its descendants, rank tokens by accumulated attention mass and keep the heaviest ones.

The arXiv paper “Adaptive Filtering of the KV Cache” challenges that assumption in a specific but important setting: schema-dense inputs such as nested JSON. The author argues that attention mass can behave less like a clean importance signal and more like a biased, non-stationary filter that favors structural roles over answer-bearing content.

Key points

• Attention mass is not always semantic importance. The paper treats accumulated attention as signal energy, then shows that this signal can be distorted by token roles in structured text.
• Non-content sink roles dominate the score. According to the abstract, delimiters or whitespace can carry an order of magnitude more energy than any content role.
• KEY tokens are over-retained. Structural KEY tokens are kept at roughly 1.8 times the rate of VALUE tokens, even though VALUE tokens are the ones carrying the answer in the studied setting.
• Accuracy can collapse under tight budgets. At a 5% cache budget, exact-match accuracy falls from 88% to 0% as the retained state becomes dominated by low-value structural information.
• Suppressing KEY tokens appears to be the best deployable correction. A counterfactual experiment identifies KEY suppression as the most practical filtering direction.
• The proposed fix is retraining-free. The method applies role-conditional allocation over SnapKV’s windowed score and is controlled by a single tuned hyperparameter. At sub-20% budgets, it closes 63% to 98% of the H2O gap.
• High-budget gains are modest and uncertain. At larger budgets, the method can match or slightly exceed full-cache accuracy, but the paper describes this as a small, seed-sensitive denoising effect, with only borderline significance at B=0.50 and no distinguishable effect from zero at B=0.30 over four seeds.

Why it matters

This work reframes KV cache eviction as more than a memory-saving heuristic. In structured data, the model may pay disproportionate attention to scaffolding: braces, separators, whitespace, and repeated keys. If the eviction policy follows that attention blindly, it may preserve the layout while discarding the payload.

That matters for practical long-context systems dealing with tool outputs, API responses, logs, configuration files, code-adjacent data, and retrieval pipelines that return semi-structured documents. These are exactly the cases where exact information extraction is often more important than fluent continuation.

The proposed approach is also notable for its deployment profile. Role labels are supplied by a roughly 15 MB linear role probe, which the paper describes as negligible in inference cost, and the base LLM does not need retraining. Still, the author is careful about the remaining gap: matching parser-level downstream accuracy is left open.

The broader lesson is straightforward: cache compression should not treat all high-attention tokens as equally useful. Future long-context inference systems may need to combine attention scores with structural roles and task relevance so that memory savings do not come at the expense of the very values the model is supposed to retrieve.

Source: arXiv