LLMs May Not Need Extra Confidence Training: The RL Critic Already Knows

Introduction

Knowing whether an answer is likely to be correct is essential when deploying large language models in settings that require trust, review, or selective automation. In practice, confidence estimation often becomes a separate project: teams collect correctness-labeled examples and train an additional self-evaluator or external model. That adds data requirements, compute, and another component to maintain.

The arXiv paper Free Lunch for LLMs Confidence: The RL Critic Already Knows explores a simpler possibility. When an LLM has been trained with value-based reinforcement learning, the critic already attached to the training process may contain the information needed to estimate confidence.

Key points

• Confidence and value prediction can be the same problem. Under a correctness-based outcome, confidence asks how likely the policy’s response is to be correct, given the question and the trajectory generated so far. A reinforcement-learning critic estimates the expected future outcome from that same trajectory. The paper formalizes the equivalence between the two objectives.
• Internal representations are not required. The authors compare self-estimators with external estimators of comparable capacity. An external model trained only on behavioral traces—questions, responses, and correctness labels—can estimate the policy’s confidence as accurately as the policy itself. This challenges the assumption that useful confidence must come from a model’s private internal state.
• The existing critic can be reused. For LLMs trained with value-based RL, the critic can serve as an off-the-shelf confidence module. No separate confidence-training phase or additional dataset is required, while the policy remains unchanged.
• Confidence can be available during generation. Because the critic evaluates trajectories, it can provide estimates not only after a final answer but also at intermediate reasoning steps. That could support early warnings, selective verification, human review, or decisions about whether to invoke tools.

Why it matters

The reported evaluation spans mathematics, coding, tool calling, and general reasoning across a dozen domains. The reused critic was the best confidence estimator on every in-distribution split and the best-calibrated system on 14 of 16 calibration evaluations. These results suggest that confidence may be less of an entirely new capability than a new use for an existing RL component.

The engineering appeal is straightforward. A system that already has a policy and critic could add risk-aware routing, answer verification, or review thresholds without changing the policy model. This may reduce both implementation cost and the risk that a separately trained confidence model becomes misaligned with the policy it is supposed to assess.

The scope of the claim still matters. The paper focuses on value-based RL and correctness-based outcomes, so the result should not automatically be generalized to every LLM or every training setup. Performance outside the training distribution, calibration under changing tasks, and the quality of the underlying reward signal remain important questions. A critic can only be as useful as the outcome it learned to predict.

Even with those caveats, the work offers a practical design principle: before adding another confidence model, check whether the existing RL critic is already solving the relevant prediction problem.

Source: arXiv