AKV Lets Agents Manage KV Cache Without Rewriting Their History

Long-running language-model agents accumulate a growing stream of observations, tool calls, tool responses, and actions. The resulting trajectory may eventually exceed the model’s context window, but context length is not the only problem. The key-value cache associated with that trajectory must also be stored, moved, reused, and sometimes restored. In an agent system that repeatedly interacts with tools, KV-cache I/O and storage can become a dominant serving cost.

The cost of changing a prompt prefix

Most existing context-engineering techniques reduce this burden by trimming old history or replacing it with a summary. This approach changes the prefix seen by the model. Because later hidden states depend on earlier computation, changing the prefix can invalidate cached states that would otherwise be reusable. The server then has to prefill the retained context again. Keeping more history reduces recomputation but increases memory and I/O; keeping less history can hurt task performance.

The paper proposes a different abstraction: cache engineering. Instead of rewriting the trajectory, an agent can manage which parts of the already computed cache remain active. The main operations are:

• Drop excludes selected cache entries from future attention while leaving the recorded trajectory untouched.
• Repos adjusts the positional information of retained keys so that the cache can continue to be used after cache management operations.
• Policy-driven control applies these actions continuously, for example around tool responses, rather than waiting for a context-length threshold and then summarizing everything at once.

This separation between trajectory and active cache is the central idea. The history remains available as a record, while the attention-visible working set can be kept within a more manageable budget.

Why ordinary prefix matching is not enough

Direct cache control creates a serving challenge. Two requests can contain identical token sequences but end up with different cache states if they went through different sequences of drop and repos operations. A cache system that only compares textual prefixes could therefore reuse an incompatible state. Cache identity must include the relevant operation history or an equivalent representation of the resulting state.

AKV addresses this issue through a co-designed serving engine. Its external interface remains stateless and compatible with common serving APIs, while its internal implementation manages stateful cache reuse, eviction, and restoration. This design aims to preserve deployment compatibility without treating cache management as a purely textual prompt operation.

Reported results and broader implications

In the reported experiments, a rolling-drop policy based on tool responses improved average benchmark scores by 1.7 points and reduced average cost by 27.7% compared with threshold-triggered summarization. With AKV, output throughput reached up to 1.2 times the baseline, while P95 request latency improved by up to 1.5 times. The authors also used recursive self-improvement to refine cache policies, reducing cost by a further 7.1% relative to an expert-designed policy.

The broader message is that an agent’s working memory does not have to be identical to its full textual history. KV cache can be treated as a runtime resource that the agent or serving system actively manages. Tool boundaries offer practical moments for deciding what should remain visible to attention.

The approach does not eliminate the need for policy design. Dropping too aggressively may remove information needed later, and positional adjustments may interact with model behavior. The results therefore point to a promising serving direction rather than a universal policy. AKV’s code is available from the project repository, enabling further evaluation across models and agent workloads.

Source: Hugging Face Daily Papers