Complex Agentic Memory Is Not Always Necessary: Raw-RAG Revisited

Introduction

Long-term conversation understanding has encouraged researchers to build increasingly sophisticated memory systems. These systems may summarize dialogue, mutate stored memories, organize information into structures, or learn specialized retrieval policies. Such mechanisms can make memory more compact and easier to operate, but every transformation also creates a chance to omit context or alter the original evidence.

A new study from Xi’an Jiaotong University examines whether all of that complexity is necessary. Its baseline, Raw-RAG, keeps the conversation history in raw form. It uses dense retrieval and reranking to select relevant passages, then supplies those passages to a reader language model.

Main findings

The authors evaluate Raw-RAG on LoCoMo, LongMemEval, MEME, and MemoryAgentBench. Across seven benchmark-model settings, it ranks first in six. Its advantage over the strongest compared agentic memory system ranges from 0.13 to 5.53 points.

The result should not be read as proof that structured or agentic memory has no role. Rather, it shows that these mechanisms should not automatically be treated as prerequisites for long-term dialogue understanding. A relatively simple raw-history retrieval pipeline is already a powerful baseline under the tested conditions.

Why can raw history work so well?

The study points to evidence coverage as a central explanation. A transformed memory store may be concise and easier for a model to use, yet fail to retain every piece of information needed for an answer. Raw history contains more noise, but it also has a better chance of preserving both the answer value and the context surrounding it.

That context can be crucial. Useful evidence is not limited to a fact or a number. It may include relations between people, the order of events, or dependency paths that allow a model to derive an answer across multiple turns. Retrieving a final-looking statement without the relations that support it can leave the model unable to reason correctly.

Raw-RAG may therefore trade utilization efficiency for broader coverage. Even if a model can access a transformed memory store completely, the study reports an accuracy gap compared with access to raw evidence. This finding emphasizes that memory design is not only an information-compression problem; it is also a problem of preserving recoverable evidence.

Implications and limits

The practical implication is straightforward: raw-history retrieval with reranking deserves to be a default comparison point for new memory architectures. It is easier to reproduce, and it can reveal whether a more elaborate system provides gains beyond a strong retrieval baseline.

The findings do not establish that every real-world application should avoid structured memory. Storage costs, privacy controls, update latency, and very long histories may still justify additional organization. A promising design direction may be hybrid: compress and index memories for efficiency, while retaining links to the original dialogue so that important claims and reasoning paths remain auditable.

Source: Hugging Face Daily Papers