S³Gym Tests Whether LLMs Can Turn Self-Evaluation into Real Improvement

Introduction

Large language models are increasingly used as agents that act in games, tools, and other external environments. These interactions generate behavioral traces, but collecting experience is not the same as learning from it. An agent may observe a successful action without understanding why it worked, when it should be repeated, or how it should be adapted to a new state.

S³Gym is designed to examine this gap. Rather than treating an agent as a fixed policy and measuring only its immediate task score, the benchmark asks whether the agent can complete a three-part loop: self-testing, self-judging, and self-improvement.

Key findings

• Exploration and evaluation are separated. Agents can explore under permissive conditions, while performance is tested on strict held-out settings. This makes it harder to obtain gains simply by memorizing previously seen trajectories.
• The benchmark uses seven text-based games. Each environment includes an executable verifier, providing an explicit way to determine whether an action or sequence of actions succeeded.
• Three experience pathways are compared. History ICL places interaction records directly in context. Summary Memory compresses experience into score-conditioned summaries. Parameter Training attempts to encode the lessons into model parameters.
• There is no universal best method. Context-level experience improves some model–game pairs, but the best pathway changes with the structure of the task.
• Compression can discard useful detail. Summaries are helpful when experience can be expressed as reusable strategic rules. They can underperform raw history when success depends on precise, state-contingent information.
• Training is powerful but unstable. Parameter updates produce substantial gains on some tasks, yet they can also lead to unstable improvement and severe negative transfer on others.

Why it matters

The central lesson is that recognizing successful behavior is not equivalent to acquiring a reusable policy. To improve reliably, an agent must connect an outcome with the conditions that produced it and then apply that understanding in a different situation. This is a stronger requirement than remembering a high-scoring example or generating a plausible post-hoc explanation.

For agent developers, the results argue against treating longer context windows, automatic summaries, or additional training as universally effective solutions. The right representation depends on the information structure of the environment. Rule-like experiences may benefit from compression; detail-sensitive tasks may require preserving the original trace; and parameter training needs safeguards against forgetting and cross-task interference.

For evaluation research, S³Gym offers a useful distinction between memorizing success and learning to generalize it. Future self-improving-agent benchmarks may need to measure not only final scores, but also exploration quality, judgment reliability, held-out performance, and transfer across tasks. Without that full loop, self-testing and self-judging may remain descriptions of behavior rather than mechanisms for sustained improvement.

Source: Hugging Face Daily Papers