Multi-View Self-Verification Helps Vision-Language Models Know When to Rethink

Introduction

Vision-language models (VLMs) have become increasingly capable at image question answering and multimodal reasoning. Yet their fluent output is not a guarantee of correctness. A model may produce a plausible explanation while missing a visual detail, making an invalid inference, or confidently selecting the wrong answer. Without an external judge, deciding whether to trust that output is difficult.

The paper When to Rethink presents MOTIVE, a framework designed to answer a practical question: should the model accept its current answer, or spend more computation rethinking it?

A three-stage reasoning loop

MOTIVE follows an “answer, verify, accept or rethink” pipeline:

• Answer: The VLM generates a candidate answer from the image and question.
• Verify: The same backbone evaluates the candidate through five complementary verification perspectives.
• Decide: A learned reliability score determines whether the answer is returned directly or revised using verification feedback and interaction history.

The approach does not require an external judge at inference time. Instead, answer generation and verification are handled by the same backbone VLM. Its main contribution is not simply asking the model to check itself more often, but learning a reliability estimate that is aligned with answer correctness through correctness-grounded multi-view verification training.

Why multiple views matter

The authors’ analysis highlights two limitations of conventional self-verification. First, verifier capability affects judgment quality: stronger verifiers tend to provide more dependable assessments. Second, verification is highly sensitive to prompt design. No single verification prompt consistently performs best across all tasks.

This finding makes a fixed checking instruction a fragile foundation for reliability estimation. A prompt may be effective for one type of visual reasoning while overlooking another. MOTIVE responds by combining complementary verification perspectives and aggregating them into a learned score. The score acts as a gate: answers assessed as reliable are accepted, while uncertain answers trigger history-guided rethinking.

Reported results

The study evaluates MOTIVE on five multimodal benchmarks and three VLM backbones. In the results included in the source material, the average accuracy of Qwen2.5-VL-3B increases from 55.11% to 63.13%, a gain of 8.02 percentage points. For Qwen2.5-VL-7B, accuracy rises from 60.49% to 68.63%, an improvement of 8.14 points. The method also reduces the average number of answer-generation turns compared with conventional self-verification.

The broader significance is that self-correction becomes a selective decision problem rather than an instruction to rethink everything. Always adding reflection can increase inference cost and may cause a model to reinforce its own mistakes. A reliability-guided gate offers a more targeted trade-off between accuracy and efficiency.

MOTIVE does not eliminate the need for careful evaluation. Reliability estimates can still depend on training supervision, verification views, and task distribution. Nevertheless, the work offers a useful direction for building multimodal systems that are not only capable of producing answers, but also better calibrated about when those answers deserve trust.

Source: Hugging Face Daily Papers