StateM Shows How Harness Scaling Can Push Agents to 95.3%

Introduction

Long-horizon agents can fail even when their models are capable of solving the individual steps in a task. The difficulty often appears between those steps: the environment changes, a useful lesson from an earlier run is not recalled, a known procedure is skipped, or the agent decides that the task is complete too soon. The StateM paper addresses this problem by scaling the harness around the model instead of changing the model’s weights.

Core ideas

• Persistent execution state. StateM organizes work around durable states and keeps context local to each phase. This is intended to reduce confusion caused by long, mutable trajectories.
• Verified transitions and recovery. The runtime uses checked transitions and recoverable runbooks. Actions are therefore tied to explicit execution conditions rather than treated as an unstructured stream of model outputs.
• Procedures that remain visible. Postmortem findings are converted into versioned procedural practices and golden rules. Both the agent and the user can inspect these controls, making operational knowledge more explicit.
• Strong benchmark results. On Terminal-Bench 2.1, StateM lifts GPT-5.5 xhigh from an 83.1% reference score to 92.1%. The same runbook transfers unchanged to GPT-5.6. With GPT-5.6 Sol xhigh, the system reports 95.3% raw accuracy over 445 trials and at least one success on all 89 tasks.
• Transfer beyond one model. A frozen profile improves GPT-5.6 Luna from 76.7% to 85.4%. Using the same runtime, runbook structure, and golden rules, DeepSeek-V4 Flash rises from 82.7% to 88.1% under standard timeouts, and reaches 89.1% on an 88-task common core.
• Reported cost difference. The paper puts final-score API usage at about $15 for the StateM run versus $574.68 for the GPT reference, while total DeepSeek spending is reported as $52.22. These figures describe the paper’s experimental configuration, not a universal deployment price.

Why it matters

The main contribution is a shift in what counts as scaling. Instead of assuming that reliability must come from a larger model or additional training, StateM decomposes it into state management, procedural control, recovery, and experience reuse. A runtime that captures these elements can potentially be reused across models, reducing the need to rebuild an agent stack whenever the underlying model changes.

The BusinessBench results add an important qualification. Family-specific runbooks built from development sets produce held-out gains of 0.55 macro points and 1.34 micro points. Two mechanism-matched families improve by 10.04 points. In other words, concrete rules generalize when tasks share an execution structure; simply copying rules across unrelated tasks is less compelling.

The 95.3% result should still be read as a result under a particular benchmark and experimental setup, not as proof of general-purpose autonomy. StateM is better understood as an auditable execution layer that turns lessons from failures into explicit, checkable, and versioned controls. As agents take on longer workflows, this layer may become as important as the model that generates the actions.

Source: Hugging Face Daily Papers