VABench Tests Whether Multimodal Models Can Truly Act in Space

Introduction

A robot operating in the physical world must do more than identify where an object is. If an object is hidden, a viewpoint is insufficient, or the layout changes, the system must determine what evidence is missing, acquire it actively, express observations in a common spatial frame, and convert its reasoning into precise actions. VABench is designed to evaluate this complete loop of observing, reasoning, acting, and revising.

What VABench evaluates

The benchmark asks general-purpose multimodal language models to infer procedural context from RGB demonstrations. Models can select camera viewpoints, output metric Cartesian targets, and revise their commands using execution feedback. They are not given privileged object poses, oracle trajectories, or learned action heads. Instead, a fixed model-agnostic controller executes only the targets specified by the model.

VABench includes 14 base task families: 11 single-arm families and three dual-arm families. It also contains seven held-out geometry or layout variants and a long-horizon composition track involving five objects. For each base task, the study uses the same 20 physically verified seeds and evaluates 12 primary model conditions across three independent runs. In addition to terminal success, it reports nine trajectory-level behavioral diagnostics and subtask progress.

Main findings

• Local visual competence does not automatically translate into embodied competence. In the annotated run, the best model achieves 100.0% on target localization and 78.9% on spatial relations, yet its three-run macro-average task success is only 53.93%±3.17%.
• Active perception makes a substantial difference. In one matched comparison, success increases from 27.86% with passive multi-view observation to 57.50% when the model can actively control the camera. The result suggests that choosing the right evidence can matter more than simply receiving additional views.
• Geometric transfer remains fragile. On held-out geometry and layout variants, task success can fall by more than 30 percentage points, indicating that models still rely heavily on familiar spatial configurations.
• Long-horizon composition is particularly difficult. Models make meaningful partial progress, but none completes a strict long-horizon episode involving the full sequence of subtasks.

Why it matters

VABench moves spatial intelligence evaluation beyond static visual question answering. It tests evidence acquisition, coordinate alignment, metric control, and correction after failure—capabilities that are central to practical robotics. Recognizing an object correctly is not enough to guarantee that a robot can grasp, place, or compose several objects when the scene differs from prior examples.

The results also suggest that progress will require more than better image descriptions or larger language models. Embodied systems need stronger active-vision policies, more reliable spatial representations, and control mechanisms that can use execution feedback to revise plans. For researchers, VABench offers a more informative diagnostic framework than a single final success number: it can help distinguish failures of perception, spatial reasoning, action precision, and recovery.

Source: Hugging Face Daily Papers