MemT-1.0 Makes Memory a Native Capability of Foundation Models

Introduction

Foundation models usually handle memory in one of two ways. Information can be absorbed into parameters, which is expensive and difficult to update continuously, or it can remain outside the model in prompts, retrieval stores, and attention caches. The second approach is flexible, but long histories increase compute and storage costs. MemT-1.0 proposes a different path: memory should be part of the model’s computation itself, allowing information to accumulate across interactions in an internal state.

What the paper introduces

• Token-level memory updates. Built on Metis, MemT-1.0 uses Gated DeltaNet-2 to update memory within each turn. The design aims to maintain a stable recurrent state instead of repeatedly concatenating every past interaction into the prompt.
• Mixture-of-Storage. The paper introduces this mechanism to scale native memory capacity. The available abstract does not specify every implementation detail, but the underlying objective is clear: support longer-lived information without keeping all history as full attention KV entries.
• A two-stage training pipeline. After deliberate backbone preparation, the model is trained through continued pretraining followed by post-training. The process uses approximately 34 billion training tokens and is supported by dedicated training and inference infrastructure.
• Two model variants. The released family includes MemT-1.0-Nano and MemT-1.0-Flash. According to the reported evaluation, they outperform parametric-memory baselines on average, remain competitive with retrieval-augmented methods, and largely retain general capabilities.
• More efficient long histories. With a 128K-token history, MemT-1.0-Nano is reported to achieve roughly a 1.86x end-to-end inference speedup and a 66x reduction in persistent storage compared with a full-context KV cache. These gains come from maintaining a compact, continuously updated memory state rather than merely extending the context window.

Why it matters

The significance of MemT-1.0 lies in changing the relationship between a model and its past. Retrieval-augmented generation treats memory primarily as information to look up, while parametric memory treats it as knowledge written into weights. Native memory aims for an intermediate but more integrated behavior: the model can absorb useful interaction history and reuse it during later computation. This could be valuable for persistent assistants, personalized systems, and agents that need to build procedural experience across multiple steps. The paper also reports improved success on agentic tasks after retrieved procedural experience is internalized.

The approach is not without open questions. A practical memory system must determine what to write, how to correct stale information, how to forget safely, and how to prevent errors from accumulating in the recurrent state. Memory isolation across users and tasks is another deployment concern. Based on the supplied material, MemT-1.0 should be viewed as an engineering step toward memory-native models, not as a complete solution to long-term memory.

Source: Hugging Face Daily Papers