Cortex Turns Context Gains into Mergeable Parametric Memory

Introduction

Large language models can often perform better when given a relevant document, instruction, example, or behavioral specification in the prompt. The improvement is usually temporary: once the context disappears, the model returns to its previous behavior. Editing the parameters can make the change persistent, but it also risks overwriting existing knowledge or degrading capabilities that were not part of the requested update.

A paper from ByteDance Seed frames this problem as memory consolidation. The goal is not merely to insert a fact into a model. It is to internalize the useful improvement caused by context while preserving performance on non-target tasks. This framing connects context-specified editing with skill learning and behavioral steering under a shared objective.

How Cortex works

Cortex first generates prospective queries. These queries are intended to anticipate the ways in which the supplied context might be used in future interactions, rather than relying only on the original prompt. The system then observes the model’s outputs when those queries are answered with the context available. Those context-conditioned behaviors become the target for distillation into parameter changes.

The paper studies two update families. Low-rank updates offer a compact, mergeable way to add changes to an existing model. Structural updates modify feed-forward components in a more organized form, aiming to balance expressiveness with interference control. In both cases, the update is designed to preserve the distinction between the requested behavior and unrelated capabilities.

Evaluation and results

The authors introduce a five-task-family testbed and evaluate single, batch, and sequential updates. The experiments span dense and mixture-of-experts backbones from 4B to 120B parameters, as well as text-only and multimodal models. According to the paper, Cortex reaches the empirical consolidation–retention Pareto frontier in most evaluated settings. In other words, its advantage is not simply stronger writing of new information; it also lies in limiting the cost of that writing.

A highlighted experiment applies 50 sequential factual edits to a 14B model. Cortex-Structural recovers 88.9% of the gain induced by context while retaining 99.8% of the original non-target performance. These figures are meaningful evidence for resistance to cumulative interference under the paper’s protocol, but they should not be read as a guarantee for every deployment environment or editing workload.

Why it matters

Cortex suggests a middle path between repeatedly supplying long context and retraining a complete model. New information or behaviors can be converted into modules that are merged into the model, which could support continual learning and targeted adaptation. Important open questions remain: how robustly prospective queries cover real future use, how conflicting edits are resolved, and how safely an incorrect fact can be removed after consolidation. The work’s broader contribution is to treat parameter editing as preservation of context-derived gains, not just replacement of a single piece of knowledge.

Source: Hugging Face Daily Papers