Why LLMs Answer Impossible Questions: Recognition Is Not Refusal

Introduction

When a model is asked to compute cot(-540°) or evaluate (1).startswith("1"), the issue may not simply be that it made a calculation mistake. Depending on the mathematical convention or programming semantics, the prompt can be structurally invalid or lack an admissible answer. Yet language models often produce a polished, confident response. A paper featured by Hugging Face Daily Papers examines whether this behavior comes from failing to recognize the problem or from failing to route recognition into abstention.

Key findings

• Unanswerability is represented internally. Across instruction-tuned models ranging from 1.7B to 70B parameters, the researchers found a linear hidden-state direction that separates answerable prompts from structurally impossible math and code prompts. This indicates that the model forms a usable representation of impossibility before it starts generating.
• Recognition is not the same as safety refusal. The recognition direction is nearly orthogonal to the canonical refusal direction associated with trained harmful-content refusal. A model’s ability to reject unsafe requests therefore does not automatically generalize to invalid definitions, impossible operations, or malformed premises.
• Behavioral invalidity is only partly aligned. A direction defined from the model’s invalidity-aware behavior is closer to the recognition direction than the standard safety-refusal direction, but the alignment remains incomplete. The result points to an additional translation step between an internal judgment and an observable response.
• Steering affects behavior. Generation-time intervention along the recognition direction changes invalidity-aware behavior in both directions and in proportion to steering strength. Random directions do not produce the same pattern, supporting a functional role for the discovered signal.
• The geometry predates instruction tuning. Comparisons between base and instruction-tuned models indicate that the low-cosine geometry is already present at the pretraining endpoint. Instruction tuning may shape how the signal is expressed, but it does not appear to create a fully shared recognition-and-refusal pathway.

Why it matters

The paper reframes a familiar failure mode. If a model simply lacked the ability to detect an impossible prompt, the obvious remedies would be more data, stronger reasoning, or larger models. But if the model already detects the issue and fails to act on that detection, scaling alone may not solve the problem. Training and inference systems may instead need an explicit bridge from structural validity checks to response policies: abstain, ask for clarification, or explain why no admissible answer exists.

The result also has implications for evaluation. Accuracy alone cannot reveal whether a model recognized that a prompt was invalid before producing an answer. Future tests could separately measure latent recognition, the stability of that signal, and its ability to influence generation. More broadly, refusal should not be treated as one universal capability. Safety refusal, mathematical invalidity detection, and code-level type or semantic checking may depend on distinct internal pathways that need deliberate alignment.

Source: Hugging Face Daily Papers