LowRAM Exploits Key–Value Predictability to Compress the KV Cache

Introduction

Autoregressive models repeatedly attend to the representations of earlier tokens. During decoding, those representations are kept in the KV cache, whose memory footprint grows linearly with context length. This makes long-context serving expensive and limits both batch size and throughput. Existing post-training methods typically reduce redundancy across tokens, layers, or feature dimensions, or store the cache at lower numerical precision.

LowRAM explores a different source of redundancy. Its central observation is that keys and values are not necessarily independent: within an attention head, a token’s value can often be predicted from its corresponding key. The paper turns this relationship into a test-time, composable compression pipeline rather than treating keys and values identically.

How the method works

• Low-rank keys: LowRAM applies cross-layer low-rank factorisation to reduce the amount of key data that must be stored. This targets correlations across layers instead of relying only on compression within an individual layer.
• Amortised values: A lightweight predictor is fitted for each attention head. It takes keys as input and reconstructs values, allowing the system to omit full value storage for tokens that are easy to predict.
• Residuals for difficult tokens: The method retains residual corrections only for tokens with the largest reconstruction errors. Storage is therefore focused on the cases where prediction is least reliable. The reported result is 4× value-cache compression with near-lossless quality.
• Test-time adaptation: Predictors can be initialised through meta-learning to reduce prefill overhead. In workloads that permit a one-off fitting cost, such as multi-query inference, they can also be fitted from scratch.
• Selective reconstruction: At decode time, LowRAM reconstructs only a selected subset of KV pairs rather than fully restoring the entire cache. This is intended to keep the additional compute small. At 4× KV compression and a 262K context, the paper reports up to 5× higher peak decode throughput.

Why it matters

The main contribution is an asymmetric view of cache compression. Keys and values have different roles and structural properties, so they can be compressed with different mechanisms. This creates a complementary axis alongside quantisation and eviction instead of replacing them. In the reported combined setting, LowRAM reaches 26.9× compression while retaining 94.5% of baseline performance.

The approach is particularly relevant to long-document question answering and services that answer multiple queries over a shared context, where an adapted predictor can be reused. It is not a free reduction in cost: fitting predictors, selecting residuals, and performing reconstruction add implementation and runtime considerations. Actual gains will depend on hardware, context length, and workload patterns. Still, LowRAM suggests that future KV-cache systems can exploit learned relationships between cache components, not just lower precision or remove entries.

Source: Hugging Face Daily Papers