Deep Fusion Memory Brings Chunk-Level Knowledge into LLM Hidden States

Introduction

Large language models often need knowledge that is not reliably stored in their pretrained parameters. Continued pretraining can introduce catastrophic forgetting, while retrieval-augmented generation adds work during inference: a system must search a corpus, select evidence, and pass that evidence to the model. Parametric memory offers another option by encoding external knowledge ahead of time, but existing approaches can be expensive to prepare and may fuse memory only at the output level.

The paper Deep Fusion Memory: A Chunk-Level Parametric Memory for Large Language Models presents DFM, a design that treats a text chunk as the basic memory unit and injects the resulting representation into intermediate model states.

How DFM works

• Chunk-level supervision. Instead of constructing a separate retrieval target for every token, DFM uses one predicted memory representation across the tokens in a chunk. This reduces the amount of offline search needed to prepare training data.
• Causal memory prediction. A memory predictor uses preceding context to generate the memory needed for the current chunk. Retrieved chunk embeddings from the corpus provide the training signal.
• Deep fusion. The predicted memory enters intermediate hidden states through gated cross-attention. This is intended to give the memory a deeper role in the model’s computation than interpolation performed only at the final output.
• Frozen backbone training. The memory predictor and fusion mechanism are trained end to end while the original language-model backbone remains frozen, limiting changes to the pretrained model itself.
• No explicit retrieval during inference. After training, the system generates memory from context rather than querying an external index for every request.

Reported results and implications

According to the paper’s abstract, DFM improves over the strongest baseline by 12.71% on Mistral-7B-v0.3 and 11.19% on Qwen3-8B-Base across six QA tasks. In domain adaptation experiments, it reports average gains of 6.51 and 4.38 points over Qwen3-4B-Base and Qwen3-8B-Base, respectively. The proposed chunk-level construction also reduces the number of retrieval queries by 32 times and the number of retrieved items by 2048 times when preparing training data, as reported by the authors.

The broader idea is important because it combines two goals that are often in tension: lowering the cost of building parametric memory and allowing that memory to influence internal processing. Chunk-level targets may be a practical compromise between token-level precision and corpus-scale preparation costs.

Still, the supplied material does not establish how DFM behaves when knowledge changes, when memories conflict, or when the predictor generates an incorrect representation. A useful next comparison would place DFM and online RAG side by side on latency, updateability, factual reliability, and memory construction cost. DFM is therefore best viewed as a promising direction for knowledge injection, not as a universal replacement for retrieval.

Source: Hugging Face Daily Papers