World Action Agent Lets Vision-Language Models Rehearse Before Robots Act

Introduction

Vision-language models bring broad semantic knowledge and spatial reasoning to robotics, but turning that knowledge into reliable physical action remains difficult. Many existing systems use a VLM indirectly: it may predict constraints, generate a program, or select actions from a static view of the scene. The gap between seeing a situation and acting within it still has to be handled by a separate control stack.

World Action Agent (WAA) proposes to make that interface an interactive world for action. Rather than requiring a VLM to output a complete manipulation sequence at once, WAA gives it multiple agents and basic tools inside a visual action workspace. Decisions can then be observed, rehearsed, revised, and executed in a connected loop.

How the workspace works

• Contact views: The system automatically selects views around the current interaction using scene geometry. When the robot is about to grasp, push, or touch an object, the model receives a view focused on the relevant contact rather than relying only on a fixed global image.
• Action rehearsal: Each action first becomes an editable proposal. The agent can preview and revise it directly, or ask an Imagination Agent to anticipate the result and use planning feedback to improve the proposal before execution.
• In-view correction: If low-level execution leaves a residual offset, the agent can correct it from the same view in which the error was observed. Observation, rehearsal, and control therefore remain tied to one visual reference frame.

The same workspace is also used to accumulate embodied procedural knowledge. WAA can evolve multimodal skills from expert videos and human demonstrations, review them against evidence, and retrieve them through a Skill Agent. Its interaction traces can additionally be used to train smaller VLMs to operate the same harness.

Reported results

On LIBERO-Pro, WAA reaches a 75.6% average success rate when using skills evolved only from LIBERO-90. The paper reports that this outperforms the compared end-to-end vision-language action models, code-as-policy agents, and a visual-harness baseline built with the same backbone. The evolved skills also remain effective in robosuite without further learning, indicating that the skills are not limited to a single benchmark configuration.

A second result concerns model transfer. Fine-tuning Qwen3.5-9B on traces collected through the harness increases its out-of-domain success from 1.7% to 43.3%. This suggests that the workspace is more than an execution wrapper: it provides a structured format in which a smaller model can learn how to inspect a scene, formulate an action, and respond to feedback.

Why it matters

WAA’s contribution is not simply the addition of more tools around a VLM. It reorganizes robot manipulation as a sequence of inspectable decisions. The model’s semantic reasoning is connected to low-level execution through previews, planning feedback, and visual correction. For longer-horizon tasks, rehearsing an action before committing may also limit the impact of an early mistake on later steps.

The approach offers an alternative to fully end-to-end policy learning. Instead of expecting a general model to perform all control internally, it supplies a consistent action interface in which general reasoning, reusable skills, imagination, and correction can work together. The cross-environment result further suggests that the design of the interaction interface may be as important as adding more training data.

The available results do not establish robustness in unrestricted household or industrial settings. Contact-view selection, the quality of planning feedback, the cost of evidence-based skill review, and safety boundaries for complex manipulation still require broader validation. Even so, WAA provides a useful framing: a robot agent should not merely be shown a world, but should be given a world in which it can rehearse and revise its actions.

Source: Hugging Face Daily Papers