Do LLMs Really Reason the Way Their Chain of Thought Claims?

Introduction

A chain-of-thought response is often treated as a window into how a large language model reached an answer. Yet a polished explanation is not necessarily a faithful record of the computation that produced the answer. A model may rely on one internal strategy while presenting another in text, and in some cases its written reasoning can be altered without changing the final result.

Key findings

• CIA turns faithfulness into a measurable question. The study introduces CoT-Interpretability Alignment, a metric that compares the strategy described in a model’s chain of thought with the internal strategy detected by interpretability tools. The goal is not simply to judge whether an explanation sounds plausible, but whether it corresponds to the computation the model actually appears to use.
• The gap appears across several task types. The authors evaluate CIA on two-hop question answering, hint intervention, and integer multiplication, using three language models. Alignment remains limited across the benchmark, ranging from 44.8% to 75.9%. Thus, a correct answer does not by itself establish that the accompanying reasoning was causally involved.
• Faithfulness can be optimized. The researchers then conduct post-training with a reward that combines task accuracy and a parametric faithfulness signal. Their experiments show that the models’ chain-of-thought faithfulness can be substantially improved while task accuracy is maintained or improved.

Why it matters

The paper frames chain-of-thought reliability as an empirical auditing problem rather than a matter of presentation quality. CIA does not prove that a model’s internal process is fully understood, and its findings depend on the interpretability tools used to identify internal strategies. Still, it offers a useful direction for evaluation: models should be assessed not only on whether they produce correct answers, but also on whether their stated reasons correspond to the computation behind those answers.

This distinction matters for mathematical reasoning, question answering, and systems that use explanations to support decisions. An explanation that is fluent but unfaithful can create misplaced confidence, especially when users treat it as evidence of how the model reached its conclusion. The post-training results are also notable because they suggest that faithfulness need not automatically come at the expense of capability. The broader challenge is to establish whether such alignment generalizes across models and tasks, and to determine how robust the underlying interpretability signals are.

Source: Hugging Face Daily Papers