PDRVLA Adds Physical Dynamics to Vision-Language-Action Models

Introduction

Vision-language-action (VLA) models can map camera observations and natural-language instructions to robot actions, but recognizing the current scene is not the same as predicting how it will evolve. In manipulation tasks, a robot may need to estimate how an object will move after contact, how depth relationships will change, or whether an action will make later steps impossible. PDRVLA targets this gap by adding physical-dynamics representations without requiring a video world model to generate future frames during every control step.

Core approach

The central component is a lightweight Physical Dynamics Module. It takes vision-language features as input and extracts physical tokens that condition the action-generation process. Instead of running a full video world model online, PDRVLA learns a compact representation of likely future changes, with the goal of reducing latency in the control loop.

The module is trained to predict several types of future visual features:

• RGB features, representing changes in visual appearance;
• depth features, capturing changes in spatial structure;
• optical-flow features, describing motion tendencies in the scene.

A frozen pretrained video world model supplies auxiliary supervision during training. In effect, the world model is used as a source of dynamic knowledge while the deployed policy relies on a smaller module rather than repeatedly synthesizing complete future videos.

Data and results

According to the paper, training uses publicly available data. The authors expand the supervision through human-to-robot synthesis, subtask annotation, and embodied chain-of-thought (ECoT) annotation. These additions provide information beyond low-level action labels, including task decomposition and reasoning about future states.

PDRVLA reports success rates of 99.3% on LIBERO, 89.9% on LIBERO-Plus, 93.6% on RoboTwin 2.0, and 81.5% on SimplerEnv. Ablation studies examine the contributions of world-model supervision, subtask conditioning, future object-pose supervision, and ECoT training to future prediction and action success.

Significance and limitations

The main idea is a practical compromise between two extremes. A robot need not generate a complete future video online, yet it is not forced to make decisions from a purely static observation. Instead, compact physical tokens provide an intermediate signal about what may happen next. This is particularly relevant to tasks involving contact, motion, and changing visibility.

The reported evaluation is based on simulation benchmarks, and the provided material does not establish real-robot deployment performance, computational cost, or generalization across physical environments. Questions such as robustness to sensor noise, unseen objects, and complex contact dynamics therefore remain open. Still, PDRVLA illustrates one route for transferring the predictive ability of world models into lower-latency VLA control, suggesting that future progress may depend not only on language grounding and action scale, but also on explicit anticipation of physical change.

Source: Hugging Face Daily Papers