VisSearch-Bench Shows Why Multimodal Agents Struggle with Long-Horizon Visual Search

Introduction

Multimodal agents can now call web and image tools to answer questions beyond their built-in knowledge. Yet many search benchmarks use an image only to identify an initial entity; after that, the agent can proceed almost entirely through text retrieval. Such tasks say little about whether an agent can preserve visual information and connect it to evidence gathered later.

VisSearch-Bench is designed to test that missing capability. It focuses on long-horizon visual search, where an agent must organize objects, local details, and spatial relations, then link those observations to external search results across multiple steps.

What the benchmark measures

• Broad task coverage: The benchmark includes 204 human-written tasks across seven main domains and 25 subdomains. Every task provides a reference answer, traceable evidence, and scoring rules.
• Five search settings: The tasks cover Set-wise Visual Search, Sequential Visual Search, Hybrid Wide–Deep Search, Competitive Visual Disambiguation, and Visual-Structure-Guided Search. Together, they test collection-level organization, chained visual clues, candidate comparison, entity disambiguation, and the use of spatial structure.
• Process as well as outcome: In addition to full-task accuracy, the benchmark reports a Process Score. This score examines whether individual results and subquestion answers are correct and complete, helping distinguish a reliable search trajectory from an answer that happens to be right.
• Low end-to-end success: Across 16 multimodal models, the strongest system fully solved only 26.47% of the tasks. Search access alone therefore does not resolve the underlying context-management problem.

Where agents break down

The accompanying observations point to several recurring failure modes. Set-wise Visual Search is particularly revealing. Agents may stop before checking all relevant items, continue searching after finding the needed information, or return to an earlier and incorrect guess. Some use memory or speculation to fill gaps even after retrieving relevant images. Others repeatedly search for the same few objects, consuming the image budget while making the context longer and harder to manage.

Hybrid Wide–Deep Search exposes another weakness. Agents often spend their effort on text queries that provide easy progress while leaving visual verification unfinished. As more objects and evidence accumulate, the system must remember which image depicts which object and which search result supports which claim. It then has to combine those links into one complete and internally consistent response.

Why it matters

VisSearch-Bench moves multimodal evaluation beyond one-shot image recognition. It treats visual search as an evidence-management problem: the agent must preserve observations, retrieve relevant information, check competing interpretations, and know when the search is sufficient.

For model developers, this suggests that tool-use frequency and final-answer accuracy are not enough. Systems also need better visual memory, evidence indexing, stopping criteria, and cross-step consistency checks. The Process Score is useful for diagnosis because it can reveal whether a model failed to find evidence or found it but failed to use it correctly.

The benchmark does not establish that one architecture will always dominate. Instead, it highlights a shared weakness across current multimodal agents: they can process images and text separately, but they are still unreliable at turning both into a verifiable chain of reasoning. As agents enter research, shopping, and fact-checking workflows, maintaining visual context may matter more than recognizing a single image accurately.

Source: Hugging Face Daily Papers