Learn2Play Bench Tests Whether LLM Agents Really Learn from Experience

Introduction

A language model that solves a familiar task is not necessarily an agent that can learn in a new environment. Many existing evaluations provide the rules explicitly or use tasks that resemble knowledge already present in pretraining. As a result, a high score may reflect reasoning and recall rather than the ability to update behavior from experience.

Learn2Play Bench is designed to make that distinction clearer. Proposed by researchers from the National University of Singapore, it evaluates agents through a collection of newly designed text-based games. The games use novel or counterintuitive rules, so an agent must interact with the environment, observe the consequences of its actions, and adjust its behavior over repeated attempts.

How the benchmark works

The benchmark has several useful properties:

• Repeated play: Agents can try the same environment multiple times, making it possible to observe whether performance improves with experience.
• Reproducible feedback: Game responses are controlled and paired with automatic scoring, enabling more consistent comparisons.
• Changing instances: The researchers vary game instances to examine whether an agent can apply learned knowledge beyond the exact situation in which it was acquired.
• System-level comparisons: The evaluation considers backbone models, self-evolving methods, and agent harnesses rather than treating the model alone as the complete system.

This setup creates a more demanding test than a one-shot question. An agent may start with a poor score and still demonstrate learning if it changes its behavior productively. Conversely, a strong initial reasoner may fail to improve when feedback is available.

Three findings

The first finding concerns experience retention. According to the study, keeping complete records of actions and feedback can support better learning than compressing past interactions into a short list of rules or strategies. A summary is cheaper to store and easier to read, but it can remove the sequence of events, the context around a failure, or details that seemed irrelevant at the time. For memory design, compression is therefore not automatically beneficial; the right question is whether the retained information remains useful for future decisions.

The second finding is a human-agent gap. Top-performing human players reached higher peak scores than the evaluated agents. Humans also explored a wider range of strategies and repeated actions less often. This points to a weakness that is easy to overlook in language-model evaluations: agents may generate plausible explanations while still exploring inefficiently. Effective learning requires not only interpreting feedback, but also deciding which new action is worth trying next.

The third finding is that the harness matters. With the backbone held fixed, changing the way an agent organizes history, processes feedback, and selects actions can improve performance while reducing estimated inference cost. The result reinforces a broader systems lesson: an agent’s capabilities are shaped by the workflow surrounding the model, not only by the model checkpoint itself.

Why it matters

Learn2Play Bench is valuable because it offers a controlled way to study learning curves, memory policies, exploration strategies, and transfer across related situations. It also provides a useful corrective to benchmark results that implicitly reward prior familiarity with the task.

The benchmark remains a text-game environment, so it cannot by itself establish how agents will learn in messy real-world settings. Future evaluations will need more incomplete information, changing objectives, and meaningful costs for exploration. Still, the practical message is clear: improving an agent may require better experience records, more deliberate exploration, and a stronger execution framework—not merely a larger language model.

Source: Hugging Face Daily Papers