In Hybrid Language Models, Attention Recalls and Recurrence Shapes Expression

Introduction

Hybrid language models combine attention with a fixed-size recurrent state, aiming to retain the flexible retrieval of Transformer-style attention while reducing the cost of carrying long contexts. Yet the presence of two memory channels makes a basic question difficult to answer: which channel remembers the facts, and which one determines how the model responds? A new study provides a strong separation. Attention mainly recalls what was said, while the recurrent state shapes how the model says what comes next.

How the study separates the channels

The authors introduce two cache-level interventions. With split-prefill, the model first processes a context and then generates from either the KV cache alone or the recurrent state alone. This reveals which capabilities survive when one channel is removed. With state-swap, the KV cache from one context is paired with the recurrent state from another in a single forward pass. This makes it possible to test not only correlation, but also the causal contribution of each channel.

Across Qwen3.5 and Falcon-H1, the findings show a pronounced division of labor:

• Exact retrieval follows attention. When the task requires recalling specific items from the context, KV-only generation retains roughly 64%–98% of the full model’s accuracy. Recurrent-only generation collapses to zero on this type of retrieval.
• Output language follows recurrence. The recurrent state preserves about 70%–80% of the model’s language performance, while KV-only generation falls to roughly 1% language accuracy.
• Persona is also carried mainly by recurrence. Keeping the recurrent state produces persona performance around 3–5 times higher than the KV-only condition.

The state-swap experiments reinforce the conclusion: the answer’s factual value follows the KV side, while the language of the answer follows the recurrent side. One channel largely answers “what should be said”; the other influences “how it should be said.”

Beyond verbatim memory

Recurrent-only generation also accepts words that never appeared in the context when they share meaning or parts with items that did appear. This suggests that the recurrent state is not simply storing a literal transcript. It may preserve more abstract semantic, stylistic, or generative information. The reported evidence, however, does not establish that recurrence provides unrestricted conceptual reasoning.

Why it matters

The distinction has practical implications for architecture and inference. Applications requiring exact quotation or lookup from a long context still need the KV cache. Applications focused on maintaining language, style, or role may benefit more from the compact recurrent state. Future systems could therefore allocate or compress the two forms of memory according to task requirements rather than treating them as redundant caches.

The work also suggests a broader evaluation principle: hybrid models should not be judged only by whether their final answers are correct. Tests should separately measure factual retrieval, language selection, and persona retention. In this view, memory has two layers: attention preserves what was said, while recurrence shapes the next act of expression.

arXiv