Solving the Task Is Not Enough: Reasoning Models Show Weak Systematicity

Introduction

Reasoning models are increasingly capable of producing correct answers to difficult prompts. Yet success on one problem does not necessarily mean that a model has learned the structure of the problem class. A paper by Simon Schug and Brenden M. Lake, published through Princeton University, examines this distinction in “Thought without systematicity? Evaluating reasoning models on rule induction tasks.”

The study focuses on systematicity, a familiar idea in cognitive science. In broad terms, a system that understands one concept should also be able to understand closely related versions of that concept. If the underlying structure remains unchanged while the symbols, objects, or combinations are altered, a systematic reasoner should preserve its performance. For AI systems, this property is important because it separates transferable reasoning from reliance on memorized surface patterns.

Key findings

• Rule induction as the test bed. The researchers build on established cognitive-science tasks designed to probe the discovery and application of rules, rather than evaluating only standard question-answering formats.
• Equivalent variants are generated deliberately. Each task family has a compositional structure. The study uses task isomorphisms, including recombination and substitution, to create problems that differ in presentation but preserve the relevant structure.
• Correctness is not consistently preserved. Models may solve an original task and still fail on a structurally equivalent version. This gap suggests that a correct answer is not always evidence that the underlying rule has been represented in a reusable form.
• Single-form benchmarks can overstate capability. If an evaluation presents only a narrow set of surface forms, a model may benefit from familiar wording or arrangements and appear more robust than it really is.

Why it matters

The paper does not simply conclude that reasoning models cannot reason. Instead, it proposes a stricter criterion: reasoning should remain stable when a task undergoes changes that preserve its structure. This has direct implications for evaluation design. Accuracy on one version should be complemented by consistency across systematic recombinations, substitutions, and other equivalent transformations.

For model developers, the findings raise the possibility that current systems remain sensitive to context, symbols, and task templates. Potential responses include training on systematically varied examples, adding consistency objectives across equivalent tasks, and reporting transfer performance rather than only aggregate accuracy.

The broader lesson is that longer chains of thought, or a correct final answer, do not by themselves establish a robust cognitive capability. A stronger claim becomes possible only when a model can apply the same rule to novel situations that preserve the original structure. Systematicity therefore offers a useful lens for distinguishing genuine compositional generalization from success tied to a particular benchmark format.

Source: Hugging Face Daily Papers