SpireBench Tests Long-Horizon Strategy in AI Agents Through Slay the Spire

Introduction

Language models can now use tools, decompose tasks, and carry out many actions in sequence. That does not necessarily mean they can make sound decisions when the consequences arrive much later. A paper on SpireBench proposes a demanding test of this distinction by placing agents inside the complete commercial deck-building game Slay the Spire. The game forces players to balance routes, cards, relics, combat, and rest, while choices made early in a run can shape its outcome much later.

Key findings

• A complete game rather than isolated episodes. The harness exposes models to the original game's full progression instead of reducing the task to individual fights or simplified turns. The study compares 20 language models with heuristic baselines and human performance at the highest difficulty.
• A strong start, followed by decline. The best models win the first act at rates comparable to experts, outperform heuristic approaches, and roughly match the average player. As runs become longer, however, the gap between models and experts grows.
• Fight-level play is a major separator. The difference between stronger and weaker systems is closely related to how they handle resources, damage, and risk in each combat. Models may recognize local objectives without turning those decisions into a consistent strategy across the run.
• A bias toward safety. Models commonly preserve health instead of spending it when doing so could produce a stronger deck or better long-term growth. Avoiding immediate damage appears safe, but it can leave the agent underpowered in later encounters.
• Memory does not reliably produce improvement. A comprehensive, self-consistent renaming of game terms changes performance, suggesting that prior game knowledge contributes to results. Yet the memories models write between runs do not make them consistently better. Those memories are often incorrect and can reinforce excessive caution.

Why it matters

SpireBench is useful not simply because it adds another game to the benchmark ecosystem. It offers a way to separate several sources of agent behavior: stored knowledge, local reasoning, risk assessment, resource management, and luck. A long run makes these factors interact, allowing researchers to ask whether an agent failed in one battle or made an earlier decision whose cost accumulated over time.

The results also challenge short, episodic evaluations. An agent may produce a plausible explanation and still fail to manage delayed consequences. A conservative route choice can look reasonable for many turns before its strategic cost becomes visible. Future evaluations may therefore need more than final win rates: stage-by-stage success, combat-level decisions, and the actual value of experience carried across runs are all relevant signals.

The broader lesson is that long-horizon improvement may require more than larger models or longer context windows. Agents need better calibration between risk and reward, mechanisms for turning local combat experience into transferable strategy, and safeguards against storing false conclusions. SpireBench presents an unsaturated environment for studying these gaps and for identifying where long-term agentic behavior breaks down.

Source: arXiv