RoTA-Bench Exposes the 3D Reasoning Gap in Vision-Language Models

Introduction

Recognizing an image is not the same as using that image to choose a precise action. RoTA-Bench focuses on this gap. It asks whether a vision-language model can infer enough 3D structure from a 2D observation, predict how rotation will change the view, and correct its plan after receiving new visual feedback.

How the benchmark works

Each task presents a target silhouette and a 3D object seen from a rotated state. The model must select successive rotations, within a limited action budget, so that the object’s 2D projection aligns with the target. Because every action changes the next observation, the task measures more than one-shot visual matching. It also probes state tracking, action planning, and error correction.

The authors automatically construct silhouette-preserving 3D objects from 2D icons. They also use reversible initialization to ensure that the generated tasks are solvable in principle. The resulting benchmark contains 674 tasks, together with a stratified 100-task test set designed to avoid overreliance on one shape family or difficulty level.

Main findings

• The human-model gap remains substantial. GPT-5.6-Sol succeeded on only 17% of tasks, while 20 undergraduate participants collectively completed the test set four times with a 100% success rate.
• Better image overlap does not necessarily mean better pose alignment. At matched states, models more often improved silhouette overlap while worsening pose alignment: 19.4% of such cases versus 10.9% for humans.
• Fine control depends heavily on interaction history. When history was removed, 96.8% of model actions fell back to a coarse 30-degree step size, and task success dropped sharply.

Why it matters

The results suggest that current vision-language models may compare images effectively without maintaining a reliable 3D representation of geometry, orientation, and action consequences. A model can make the projection look more similar while moving farther from the correct physical pose. Static image similarity therefore provides an incomplete picture of spatial competence.

RoTA-Bench is useful because it places perception, memory, planning, and control in one feedback loop. It points toward evaluations that include action limits, intermediate observations, and precision requirements rather than only recognition or spatial question answering. This design is especially relevant to robotics and embodied agents, where the next action must be based on an evolving estimate of the world. At the same time, the benchmark relies on controlled objects generated from 2D icons, so its results should not automatically be treated as evidence about performance in complex natural scenes.

Source: Hugging Face Daily Papers