Memory Decoder at Scale: Separating Long-Term Memory from LLM Reasoning

Introduction

Decoder-only language models usually store knowledge and perform reasoning within the same set of weights. This unified design is convenient, but it makes memory capacity difficult to scale independently. If a model needs to remember more, the common answer is to make the whole model larger. Memory Decoder at Scale: A Pretrained, Parametric Long-Term Memory explores a different route: keep the reasoning backbone relatively small, and attach a much larger pretrained memory component.

Key points

• Disentangling memory and reasoning
  The paper builds on the Memory Decoder idea, where long-term memory is treated as a separate parametric module rather than being fully entangled with the language model backbone. The base model remains responsible for generation and reasoning, while the memory module carries a larger share of stored knowledge.

• Scaling the memory module
  Earlier Memory Decoder experiments were conducted at smaller scale. This work pushes the memory model up to 6.9B parameters and pretrains it on 300B tokens, shifting the question from feasibility to whether the approach remains useful at serious training scale.

• A systems problem: large-scale retrieval
  At this data scale, a standard Faiss indexing and search workflow becomes too costly. The authors address the bottleneck with a distributed pipeline for Faiss indexing and retrieval, plus sparse batch-wise loading of kNN distributions. This is an important part of the contribution because the memory design depends on retrieval being tractable.

• Reported gains across models
  On 17 benchmarks, Pythia-410M paired with a 6.9B general memory improves its average score from 29.86 to 37.34. That slightly surpasses Pythia-12B at 37.24 while using 39% fewer total parameters. For Qwen3 Base models from 0.6B to 14B, 1.7B domain memories improve the average score across three domains by more than 9 points at every scale.

Why it matters

The most interesting implication is not just the benchmark number, but the proposed scaling pattern. Instead of spending all new parameters on the main language model, developers may allocate a substantial portion to a pretrained memory module. This suggests a “small backbone, large memory” paradigm that could be useful when parameter efficiency matters.

The approach also shifts some complexity into the infrastructure layer. Large memory models require scalable indexing, retrieval, and careful coordination with the base model. Latency, domain selection, and the interaction between general and domain-specific memories will likely determine how practical the method becomes.

This direction is also distinct from classic RAG. RAG retrieves external documents, while this work focuses on trainable parametric memory. In the long run, both ideas may be combined: external retrieval for fresh or verifiable information, and parametric memory for compact, reusable long-term knowledge.

Source: Hugging Face Daily Papers