Grouped Value Attention Cuts KV Cache Storage with On-Demand Key Reconstruction

Introduction

For long-context generation, the KV Cache can become a more immediate systems bottleneck than the model weights themselves. Every decoded token adds a key and a value to the cache, while each later decoding step must read an increasingly long history. Grouped-query attention (GQA) reduces this cost by allowing multiple query heads to share key-value heads, but it still stores both a key and a value for every position.

The paper “Grouped Value Attention: Efficient KV Caching via On-Demand Key Reconstruction” explores whether the cache can be made smaller by changing the representation rather than merely reducing the number of attention heads.

Key ideas

• Persist grouped values. GVA organizes values into groups and avoids persistently storing a complete content key in the conventional form for every position.
• Reconstruct content keys on demand. A learned linear map links the value representation to the content-key representation. During inference, the map can be absorbed into the query transformation. In the intended decode path, this removes the need to explicitly materialize content keys.
• Keep position information separate. Compressing the content representation could otherwise weaken positional modeling. GVA therefore adds a small shared decoupled RoPE channel and caches a separate positional key. The study includes a 16-dimensional positional configuration.
• Reduce persistent cache size. Across the configurations examined, the method lowers the number of persistent cache scalars by approximately 45–47% relative to matched GQA. This is a representation-level storage result, not proof of an equivalent end-to-end latency reduction.
• Remain close to GQA on the reported benchmark. At the 350M-parameter scale, trained on 30B FineWeb-Edu tokens, the 16-dimensional positional variant achieved 44.18 average accuracy across five tasks. GQA reached 44.36, while MLA reached 43.88.

Why it matters—and what remains open

GVA’s central contribution is the separation of content and positional information in the cache. By shifting part of key reconstruction to the query side, it may reduce memory footprint and cache-read traffic for long-context serving, large batches, and deployments constrained by accelerator memory. A smaller persistent representation could also make room for more concurrent requests or longer histories.

The main caveat is that fewer cached scalars do not automatically mean faster decoding. Reconstructing or transforming representations still has a compute cost, and the final outcome depends on memory access patterns, kernel fusion, batching, and hardware utilization. The authors report that custom decoding kernels have been developed and that their end-to-end inference performance is still under evaluation. An open-source release is planned, but the supplied material does not yet establish a measured speedup.

The most important follow-up questions concern scale and operating conditions. Will the approach preserve its quality gap at larger models and longer contexts? Can the kernels turn lower cache traffic into lower latency or higher throughput? And how should the positional-channel dimension be selected when quality, memory, and compute pull in different directions? For now, GVA is best viewed as a promising cache representation design with encouraging benchmark results, rather than a fully validated drop-in acceleration technique.

Source: Hugging Face Daily Papers