Beyond Accuracy: How SFT, RL and OPD Reshape Confidence in Chain-of-Thought Reasoning

Lead

Most discussions of post-training for large language models focus on whether methods such as supervised fine-tuning, reinforcement learning and on-policy distillation improve final-answer accuracy. This arXiv paper asks a more diagnostic question: how do these methods change a model’s confidence while it is reasoning?

The authors argue that confidence in chain-of-thought reasoning should not be treated as a single global score. A model may be well calibrated before it starts reasoning, poorly calibrated in the middle of a trace, and useful again near the end—or the reverse. Understanding this pattern matters for systems that must decide whether to allocate more tokens, stop early, or combine multiple reasoning paths.

Key points

• A three-stage calibration view: The paper evaluates confidence before chain-of-thought generation, during generation and after generation. These stages correspond to difficulty estimation, early termination and answer aggregation.
• Post-training methods specialize differently: In controlled mathematical reasoning benchmarks, OPD provides the most useful pre-reasoning confidence signal. SFT offers the strongest online signal for deciding when to stop early. RL produces the most reliable trace-level signal for aggregating answers across sampled reasoning paths.
• Confidence depends on position: The study finds that RL confidence becomes informative only after a path-commitment phase. OPD confidence is helpful early, but later in the reasoning process it can become inversely calibrated, meaning that higher confidence may no longer indicate a higher chance of correctness.
• PosConf uses confidence selectively: Based on this position effect, the authors introduce PosConf, a position-aware strategy that relies on confidence only in relative-position intervals where it has been shown to be reliable.

Why it matters

The work reframes evaluation of reasoning models. Accuracy tells us whether a model eventually reaches the right answer, but calibration tells us whether the model’s internal signals can be used to make better runtime decisions. That distinction is important for token-budgeted inference, multi-sample reasoning and applications where incorrect high-confidence outputs are costly.

The reported results show that PosConf improves RL answer aggregation by 6.1 points over majority voting. Under tight token budgets, it also improves OPD early stopping, with gains of up to 4.3 points by avoiding later regions where OPD confidence becomes unreliable.

The broader lesson is practical: developers should not assume that confidence is uniformly meaningful across an entire chain of thought. Post-training changes not only accuracy, but also when and where confidence aligns with correctness. Future reasoning-system design may therefore need stage-aware and position-aware calibration methods alongside conventional accuracy benchmarks.

Source: arXiv