ACT3D Teaches Vision-Language Models to Explore 3D Worlds

Introduction

A vision-language model may recognize objects in a single image yet still struggle with a 3D question when the relevant evidence is hidden, distant, or visible only from another direction. ACT3D addresses this gap by making visual reasoning interactive. Instead of asking the model to infer an entire scene from a precomputed collection of views, the system lets it decide whether the current view is sufficient and, if not, where to move next.

The central design choice is deliberately simple. The model retains the native visual interface used by many VLMs: one image at a time. A 3D representation is kept outside the model as an environment. The model can issue actions such as moving left, receive a newly rendered image, and continue the reasoning process.

How the approach works

• Question-conditioned rollouts: Each episode begins with a first-person observation and a visual question. The model must connect the question to the evidence it needs to collect.
• A perception-action loop: Reasoning, movement, reorientation, and new observations can be repeated before the model commits to an answer.
• Outcome-based reinforcement learning: After trajectory-supervised fine-tuning, the system is optimized mainly according to final-answer correctness. It is not given a label for every ideal viewpoint or exploration step.
• Learning efficient information gathering: Improvement in the final answer becomes the signal for discovering useful behavior. The model can search for occluded targets, obtain complementary views for spatial comparisons, or stop when further movement is unnecessary.

Results

Across six 3D benchmarks, ACT3D records 53.2% average accuracy. The paper reports a 22.0% improvement over its base Qwen model and performance above the compared proprietary Gemini 3.6 Flash system. It also exceeds existing open-source 3D methods while consuming 50–70% fewer input frames, suggesting that efficient viewpoint selection may matter more than simply expanding the visual context.

The gains depend strongly on the difficulty of exploration. On questions answerable from the initial view, the improvement over the base model is only 0.6%. On questions requiring extensive exploration, the gain reaches 16.3%. This pattern indicates that the policy is not merely learning to take more actions. It is learning to make exploration conditional on the question and the evidence already available. On unseen domains, ACT3D improves by an average of 5.9% over its supervised initialization, providing evidence of transfer beyond the initial training setting.

Why it matters

ACT3D frames 3D understanding as an evidence-acquisition problem. A model does not need to receive every possible view in advance; it can gather information through controlled interaction. That idea is relevant to embodied agents, navigation systems, and future robotic assistants, where sensing and acting are naturally coupled.

There are still open questions. A reward based mainly on final correctness can be sparse, especially in long-horizon environments. The paper therefore does not establish that the same strategy will remain reliable under continuous control, real-world perception noise, or open-ended tasks. Even so, its results show that reinforcement learning can teach a VLM not only to interpret visual evidence, but also to decide when additional evidence is worth acquiring.

Source: Hugging Face Daily Papers