Why Long-Horizon Agents Often Survive Context Compaction

Introduction

Long-running agents eventually face a basic systems problem: the conversation becomes larger than the model’s context window. To keep the task moving, an agent scaffold must remove or summarize older turns. This process, commonly called context compaction, extends the usable lifetime of an agent but is inherently lossy. Instructions, task constraints, and the reasoning behind earlier decisions can disappear without an obvious failure signal.

The paper Benchmarks Are Surprisingly Compaction-Friendly examines how serious that problem is in realistic, long-horizon settings. The authors fixed the OpenCode agent scaffold, added five compaction strategies, and evaluated three frontier models on PaperBench, ProgramBench, and PostTrainBench.

Main findings

• For the two models with 1M-token context windows, changing the point at which compaction begins did not measurably change benchmark performance.
• GPT-5.3-Codex, which has the smallest context window in the study, performed worse with lower compaction thresholds on PaperBench. A less pronounced pattern appeared on PostTrainBench.
• None of the alternative methods consistently beat the default approach of letting the model summarize its own context. Strategies that discard more material cost some models as much as 20 points, while leaving other models largely unchanged.
• Compaction can remove constraints even when the task still appears to succeed. On ProgramBench, the first summary dropped roughly one third of the anti-cheating rules on average, and the loss increased with repeated compaction.
• In real SWE-chat sessions, instructions about how the agent should work and what it must not do were dropped around six times more often than descriptions of what should be built.

Why the benchmarks look forgiving

The authors argue that current benchmarks are “compaction-friendly.” Their objectives are generally fixed, and much of an agent’s progress is written to disk in the form of code, documents, or other artifacts. If the conversation loses part of its history, the agent can often inspect those artifacts and reconstruct the practical state of the task.

That recovery path does not work equally well for every kind of information. Files can show what has been produced, but they may not preserve why a decision was made, which procedure the agent was required to follow, or which actions were prohibited. Such rules may appear only in an earlier prompt or exchange. Once omitted from the summary, they can be difficult to recover.

The distinction matters because task completion and rule compliance are not the same thing. A benchmark score may remain stable while the agent’s behavior becomes less faithful to the original specification.

Implications

The study does not undermine compaction. It shows that compaction can be operationally effective while remaining semantically unsafe. Evaluating only final task scores may therefore miss a class of failures in which an agent delivers a plausible artifact after forgetting important boundaries.

Agent frameworks could respond by maintaining a protected record of non-negotiable constraints, separate from ordinary conversation history. Evaluation suites should also include tasks whose rules cannot be recovered from external files and should measure process compliance, not just the final artifact. As agents take on longer workflows, preserving “how to work” may become as important as preserving “what to build.”

arXiv