Why LLMs Keep Thinking: A Study of “Thinking Inertia”

Introduction

Large language models are increasingly offered with explicit thinking modes. Users can ask them to spend more time reasoning before producing an answer, and many systems now expose controls intended to manage that behavior. The opposite capability—stopping visible reasoning when the user asks for an answer only—has received much less attention.

The paper Thinking Inertia: LLMs Keep Thinking When Told Not To examines whether models can reliably follow such instructions. Its central observation is that answer correctness and instruction compliance are separate properties. A model may produce the right answer while still adding analysis, explanation, or inference before it. The authors describe this persistence under explicit constraints as “Thinking Inertia.”

A more careful definition of no-thinking

A common way to evaluate no-thinking behavior is to use a proxy, such as disabling a thinking mode or checking whether the response is short. The paper argues that neither proxy is sufficient. A disabled mode may still allow reasoning-like text to appear, while a long response may consist of filler rather than genuine inference.

The researchers therefore normalize each response into two parts: the text before the answer and the final answer itself. They evaluate the pre-answer portion at three levels:

• Empty-Thinking Rate measures strict answer-only compliance. A response counts as successful only when it contains no pre-answer text.
• Question-Pre-answer Relevance measures how closely the pre-answer text relates to the question. This helps distinguish task-related content from generic filler.
• Explicit Inference Rate uses an LLM judge to identify whether the pre-answer text contains visible, explicit inference.

Together, these measures separate three behaviors that are often treated as the same: direct answer-only output, relevant text without clear inference, and openly displayed reasoning.

What the experiments show

The study evaluates six prompting interventions on six LLMs across Boolean questions, multiple-choice questions, and open-ended questions. The broad result is that explicit no-think controls do not reliably remove visible inference. Even when instructions are strict, models may continue to explain how they reached an answer before presenting it.

Task structure matters. On Boolean and multiple-choice tasks, accuracy remains stable while the models are asked to suppress visible reasoning. Open-ended tasks show a stronger form of Thinking Inertia. As the answer space becomes less constrained, models are more likely to produce analysis and explanation alongside the answer. The supplied material also points to a trade-off between answer performance and no-thinking compliance in open-ended settings, without providing a more specific quantitative claim.

Why this matters

The findings suggest that “thinking disabled” should not be treated as equivalent to “no reasoning text was generated.” Nor should response length alone be used to infer whether a model has stopped thinking. Applications that require concise output, strict formatting, low latency, or limited disclosure of intermediate reasoning need a more direct behavioral evaluation.

For model developers, no-thinking behavior may need dedicated training and decoding controls rather than a simple switch that reverses a thinking mode. For evaluators, the proposed three-level framework offers a way to distinguish the absence of text, the presence of relevant but non-inferential text, and the display of explicit reasoning. More broadly, a capable model should not only know how to reason; it should also know when to stop showing that reasoning.

Source: Hugging Face Daily Papers