Chain-of-Experience: Helping LLMs Improve During Inference

Introduction

Most language-model evaluations treat a prompt and its response as a complete episode. This setup measures what a model can do immediately, but it leaves out an important part of human problem solving: people learn from failed attempts, revise their approach, and try again. The paper Chain-of-Experience for Continual LLM Improvement studies this ability at inference time and names the setting Chain-of-Experience (CoE).

Key findings

• Inference becomes an experience loop. CoE does not require updating model weights during deployment. Instead, the system accumulates traces from repeated interactions and places those traces into the context of later attempts.
• Feedback can come from multiple sources. The study considers model-generated self-feedback as well as environmental signals, including answer correctness and public coding-test pass rates. These signals capture different aspects of progress.
• The evaluation spans several domains. Experiments cover mathematics, coding, and knowledge tasks, using eight LLMs. The abstract specifically names GPT-5, Gemini-2.5 Pro, and Claude-4.5 Sonnet among them.
• Iteration beats no-feedback inference. The reported results show that experience-based interaction consistently outperforms feedback-free baselines. Across tasks and models, the paper reports a 5.6% overall improvement and 19% lower API cost, while also claiming higher accuracy per token than existing test-time strategies.
• Complementary signals help. Combining model feedback with correctness-oriented environmental feedback produces additional gains. The authors also report a positive relationship between a model’s base capability and its capacity to improve, with most gains appearing early in the iteration process.

Why it matters

CoE broadens the meaning of test-time scaling. More inference compute does not have to mean simply sampling more answers or producing longer reasoning traces. It can also mean preserving the results of previous attempts and using them to change the next strategy. This is especially relevant to tasks with verifiable outcomes, such as mathematical answers and code that can be run against tests.

The framework also suggests a more dynamic evaluation question: not only whether a model is correct on its first attempt, but whether it can become more reliable after receiving useful experience. That distinction may matter for agents operating in environments where actions produce observable results.

There are important qualifications. Robustness to weak or spurious feedback does not imply that every feedback channel is safe or informative. The supplied material does not provide the detailed experimental setup, iteration budgets, or cost accounting behind the headline numbers, so the results should not be treated as a universal deployment guarantee. Practical systems will still need methods for ranking memories, detecting misleading feedback, and deciding when further iterations are no longer worthwhile.

Even with those open questions, CoE offers a useful lens for designing and evaluating models that improve through interaction rather than remaining fixed after the first response.

Hugging Face Daily Papers