See2Think asks whether multimodal models truly use intermediate visual states

Introduction

Multimodal large language models are increasingly presented as systems that can “think visually.” They may draw sketches, add annotations, call visual tools, or generate intermediate images before giving an answer. But the presence of these artifacts does not automatically prove that the model is using them. A model might produce a plausible visual trace while still relying mainly on text cues, dataset priors, or language-only reasoning.

The paper See2Think: Do Multimodal Models Really Use Intermediate Visual States? tackles this gap directly. Instead of evaluating only the final answer, it proposes a framework for inspecting the full visual reasoning process: what visual action the model chooses, how that action is rendered, and whether the subsequent reasoning actually responds to the visual state.

Key points

• A process-level evaluation framework: See2Think combines See2ThinkBench with Visual Action-of-Thought, or VAoT. VAoT records textual thoughts, visual actions, rendered states, and later reasoning, making it possible to analyze the steps behind a final response.
• Tasks designed to require vision: See2ThinkBench contains 1,200 open-ended problems across 12 categories. These span 2D structured reasoning, 3D scenes, and real-world reasoning. The emphasis is on visually dependent samples, reducing the chance that models can solve tasks from text alone.
• No universal best setting: The authors evaluate representative proprietary and open-source multimodal models under controlled inference settings. Their results suggest that visual reasoning performance depends strongly on both the model and its environment; no single configuration dominates across all tasks.
• Rendering is the clearest bottleneck: Models can often select visual operations that are relevant to the task. However, turning those intentions into faithful intermediate visual states remains difficult. In other words, knowing what to draw or annotate is not the same as producing a reliable visual artifact.
• Visual dependence can be tested by intervention: When task-relevant corrupted feedback is introduced, model accuracy drops by more than 10 percentage points in controlled interventions. This indicates that intermediate visual states can affect behavior, although high uptake of feedback does not necessarily translate into higher accuracy.

Why it matters

See2Think shifts the discussion of multimodal reasoning from outcome scoring to behavioral diagnosis. A final answer alone cannot tell whether a model reasoned from visual evidence, guessed correctly despite flawed steps, or bypassed the visual channel altogether. By tracking actions, renderings, and feedback usage, the framework offers a more granular way to evaluate claims about visual thinking.

For model builders, the results point to a practical research agenda. Improving multimodal reasoning may require not only stronger language reasoning, but also better mechanisms for generating faithful intermediate images and using them reliably. For benchmark designers, the work is a reminder that tool-augmented or visually mediated reasoning should be evaluated across the full chain, not just at the end.

In short, See2Think does not dismiss intermediate visual states. It shows that they can matter, but also that current systems remain inconsistent: they may choose the right visual operation, fail to render it faithfully, or absorb feedback without improving the final answer.

Source: Hugging Face Daily Papers