JitMem Lets LLM Agents Curate Memory Only When They Need It

Introduction

Giving a language-model agent memory is not simply a matter of saving old conversations. A memory system must decide what to preserve, how to represent it, and how to make it useful when a later task arrives. Many existing approaches make that decision at write time: after an episode ends, they distill the trajectory into a reflection, workflow, skill, or reasoning recipe. The resulting artifact is easy to retrieve, but it is created before the system knows what a future user query will ask for.

The paper Just-in-Time Memory: Learning to Curate Task-Adaptive Memory for LLM Agents challenges this design choice. Its Just-in-Time Memory approach, or JitMem, delays curation until the memory is actually needed.

Key points

• Raw experiences are retained. Instead of immediately converting every trajectory into a fixed summary, JitMem keeps the original trace available. This reduces the risk that a write-time compressor will discard details that later become useful.
• Curation is conditioned on the current task. When a new task arrives, the system retrieves potentially relevant traces and gives them, together with the task, to a memory curator. The curator then produces a compact payload designed for the immediate objective. The same past experience can therefore be presented differently for different queries.
• Training receives a shorter feedback path. In a write-time system, the value of a storage decision may not be visible until a related task appears many episodes later. JitMem’s output is used on the same task that prompted curation, allowing training to rely directly on immediate task success and avoiding the need to group related tasks artificially.
• The reported gains are consistent. On ALFWorld, WebShop, and τ^2-bench, JitMem outperforms no-memory agents and both heuristic and learned write-time memory methods. The paper reports absolute improvements over the strongest baseline of 16.2, 16.3, and 3.9 success-rate points, respectively. Even an untrained curator is competitive with, or better than, the baselines, while training adds further gains.

Why it matters

JitMem is important because it changes the timing of the central memory decision. Write-time compression favors a small and reusable archive, but it must guess the future value of information. Read-time curation separates preservation from interpretation: the system keeps experience available and decides how to organize it only after the next objective is known. This makes memory more adaptable to open-ended environments where future requests are difficult to enumerate.

The approach also has practical trade-offs. Keeping raw traces can increase storage, retrieval, and context-processing costs, and the final result still depends on retrieval quality and the agent executing the curated payload. JitMem therefore does not imply that every trajectory should be stored forever. Its broader lesson is that compression may be better treated as a demand-driven operation. For long-lived LLM agents, memory could become less like a static archive and more like a context resource reconstructed for each task.

Source: Hugging Face Daily Papers