LatentPress Lets Language Models Read Compressed Memory Directly

Introduction

Long-context inference is not only a question of fitting more tokens into a window. As conversation histories and documents grow, the model must process more input, increasing attention work, cache usage, and latency. Common compression strategies usually produce readable summaries or render information as images. Both approaches add another decoding step: the language model must interpret generated text, or a vision-capable system must reconstruct information through OCR or visual processing.

The paper LatentPress: Context Compression Beyond Text and Vision explores a different interface. Instead of representing compressed context as human-readable language, it encodes conversations and documents as continuous memory tokens. A frozen language-model decoder then reads those tokens directly through its input-embedding interface. The target of compression is therefore not a person, but the downstream model that will use the memory.

How it works

LatentPress uses a lightweight, reader-matched writer. The writer maps a long context into a shorter sequence of continuous vectors, which are passed to the frozen decoder as memory. At inference time, the system does not reconstruct the memory as text. This removes the need for a summarization pass, avoids OCR reconstruction, and keeps the compressed representation in the format most directly consumed by the language model.

The reported training setup updates only an adapter of about 4.2 million to 26.2 million parameters, or roughly 0.1% of the decoder. Depending on the configuration, the writer supports approximately 4x to 16x compression. Writing a conversation takes 43 milliseconds in the reported experiments. Reading is reported to be 5x to 9x faster than processing raw context or cached OCR, while writing is roughly an order of magnitude faster than text summarization or OCR reconstruction.

Main findings

• On LongMemEval, LatentPress reaches 0.504 accuracy at 7.70x compression, compared with 0.490 for uncompressed evidence.
• Text-summary baselines reach 0.184 accuracy, while OCR-based compression ranges from 0.426 to 0.312 across the reported settings.
• On LongBench-QA, in-domain writers match or exceed raw-context reading at 4x to 8x compression. At 16x compression, performance falls below the raw-context condition.
• Two transfer settings test zero-shot generalization: one transfers a writer from UltraChat to LongMemEval memory QA, and the other transfers from LongMemEval-derived QA to previously unseen LongBench document domains.

Why it matters

The most important contribution is an interface choice. Summaries are useful when people need to inspect compressed information, and images can provide a cross-modal carrier. But a language model already consumes learned vector representations. Continuous memory tokens may therefore offer a more direct machine-facing format for long histories, document question answering, and memory-augmented applications.

The results also define a practical boundary. Compression is not automatically beneficial at every ratio: the LongBench-QA results show that a 16x reduction can discard information that matters for answering questions. The reader-matched design may also make transfer across different decoders less straightforward. Interpretability, failure diagnosis, reuse across models, and behavior on more complex tasks remain open questions.

LatentPress should consequently be viewed as a promising context-infrastructure direction rather than a universal replacement for summaries or retrieval. Its central idea is to treat compressed context as a learned, continuous interface between a writer and a reader, potentially reducing both the cost and the number of transformations required during inference.

Source: Hugging Face Daily Papers