CLE Tests Whether LLM Agents Can Truly Learn Over Time

Introduction

Most LLM benchmarks ask whether a model can solve a task presented at a particular moment. That is useful, but it leaves out a defining property of an autonomous professional agent: the ability to learn from previous work. A capable long-horizon system should preserve useful knowledge, understand why an attempt failed, and apply the resulting lessons to later assignments. The Continual Learning Exam (CLE) is designed to measure that capability directly.

What CLE evaluates

Built with the participation of more than 50 domain experts, CLE contains 120 human-authored professional tasks. Each task is structured as a five-level progression, producing 600 levels in total across 89 occupations and 12 occupational domains. The benchmark also includes roughly 24,000 real-world files, 19 stateful applications, and more than 16,000 human-authored evaluation criteria.

The ladder structure is central to the benchmark. An agent must pass one level before accessing the next. Later levels are not merely harder versions of the same prompt; they require the system to reuse and extend knowledge, strategies, and skills acquired during earlier work. In other words, CLE tests whether an agent can turn experience into a durable capability rather than simply produce a strong isolated answer.

An evaluation agent scores submissions against human-written rubrics. The authors compare a single-submission setting with a more interactive setting in which an agent receives diagnostic feedback after failure and is allowed one revision. This creates a limited version of the execution, reflection, and correction loop that would occur in a real workflow.

Main findings

• The study evaluates 17 combinations of frontier models and agent harnesses.
• Feedback plus one revision produces a median completion gain of 5.3 percentage points.
• The best-performing system completes 47.7% of all levels.
• It completes every level in only 23.3% of the task ladders.

The results expose a gap between local competence and sustained learning. A system may perform well on an individual assignment without reliably carrying lessons into the next stage. Feedback helps, but the measured improvement is not large enough to suggest that current agents can independently build robust, reusable skills over long horizons.

Why it matters

CLE brings memory, skill reuse, experience transfer, and recursive self-improvement into a single evaluation setting. For agent researchers, that makes it possible to ask more specific questions: Does the system retain useful context? Can it diagnose a recurring error? Does a strategy learned in one assignment help with a later one? And can those gains survive as the workflow becomes more demanding?

The benchmark should not be read as a complete measure of real-world intelligence. Results can also depend on tool use, harness design, feedback quality, and the behavior of the evaluation agent. Still, the inclusion of files, stateful applications, and human-authored criteria makes CLE more representative of extended professional work than a collection of independent question-answering tasks.

CLE may also serve as training data for post-training systems aimed at better continual learning. Its broader message is straightforward: frontier LLMs are increasingly capable at solving individual problems, but reliable “learning while working” remains an unresolved engineering and research challenge.

Source: Hugging Face Daily Papers