FM-Bench Tests Whether AI Agents Can Manage a Football Club for 20 Years

Introduction

Language models can now handle many bounded tasks reliably. A harder question is whether an agent can keep making sensible decisions when the consequences arrive much later. FM-Bench, short for Football Management Benchmark, approaches that question through a 20-year club-management simulation rather than a sequence of isolated prompts.

Turning long-term management into an evaluation

Each agent runs a football club for 20 in-game years using 26 tools and roughly 340 to 400 decision stops. It must build a squad under the same budget as its rivals, trade players, negotiate contracts, invest in facilities and youth development, set lineups, and respond to a board that can dismiss it. A deterministic game engine aggregates the outcome of every season into a final score, removing the need for an LLM judge or a human evaluator.

The benchmark has two tracks. In Solo, each of 15 frontier models faces a frozen scripted world. In the Arena, the same models, joined by a scripted anchor, compete in one shared 20-year world. The researchers also examine six behavioral capabilities behind the aggregate score, aiming to distinguish genuine management patterns from one-off wins.

What the results show

• Across three seeds, all 15 models completed the full horizon, while blind scripted baselines failed in most of their runs.
• claude-fable-5 led the Solo table by mean score and also topped the Arena overall, although the Arena title rotated among ten different models.
• Model scale, price, and vendor did not predict the ranking. The ordering became clear only late in the horizon.
• Higher-scoring agents tended to reduce slow-payoff investment near the end, keep cash deployed rather than idle, and start contract renewals well before deadlines.
• Rejected bids did not lead models to reliably infer hidden market prices, even after hundreds of interactions.
• Memory management failed in two opposite ways: some systems accumulated an ever-growing archive, while others rewrote their plan every season.
• The best first-play human participant ended up at the bottom of the model leaderboard, underscoring how demanding persistent state tracking can be.

Why it matters

FM-Bench shifts attention from whether an agent can complete a task to whether it can manage a feedback loop over time. A decision that looks sensible in one season may damage future finances, weaken squad continuity, or reduce the club’s ability to compete. Long-horizon environments therefore expose planning drift, poor timing, unused resources, and weak memory practices that short benchmarks can miss.

The benchmark is not a complete proxy for real-world management. Its findings depend on the simulation’s rules, tools, and deterministic engine, and strategies learned there may not transfer directly to other domains. Still, it offers a useful template for agent evaluation: measure not only the quality of individual actions, but also whether an agent maintains a coherent objective, updates its plan from feedback, and accounts for consequences that appear much later.

Source: Hugging Face Daily Papers