Back to articles

O²LaF Uses One Pretrained Video Prior to Learn Two Dynamics

3 min read

Videos show how scenes change, but they rarely label the actions that caused those changes. For robotics and world models, this leaves a gap: visual data can reveal outcomes without providing the action supervision needed to learn how an environment behaves. Latent-action learning tries to infer hidden actions from visual transitions, but many approaches still learn much of the underlying dynamics from scratch. O²LaF takes a different route, using a pretrained video model as a starting prior for both action inference and consequence prediction.

One prior, two dynamics

O²LaF initializes two task-specific models from the same pretrained video diffusion Transformer, or DiT. Its inverse-dynamics component is a feed-forward transition encoder that uses learnable action queries to extract latent actions from changes between video states. Its forward-dynamics component is a causal, action-conditioned video generator that predicts subsequent frames. The models share an initialization source, but are adapted for distinct roles rather than making one model perform every step.

During training, the framework controls what information each component can access, keeping the modules aligned with their assigned tasks. The inferred action representation also passes through a compact bottleneck. The two dynamics are optimized end to end, so the action code must represent observed transitions while also supporting predictions about what happens next. Training uses unlabeled videos, without requiring human action annotations for each clip.

What the paper reports

  • In navigation and robotic manipulation, O²LaF is reported to achieve a new state of the art in few-shot action probing.
  • The authors report stronger zero-shot action transfer across visual contexts and embodiments.
  • The framework is also extended to predictive representation priors, including V-JEPA2, with competitive performance.

These findings suggest a broader role for pretrained video models: beyond generating frames, they may provide a useful foundation for learning the relationship between actions and environmental change. If an action representation transfers across scenes, robots could obtain knowledge from observed video that is more relevant to control than visual features alone. The forward predictor also offers a way to test whether a latent action can account for its expected consequences.

The available abstract, however, does not name benchmarks, provide numerical gains, or describe experimental settings in detail. It therefore does not show how large the reported advantages are or under which conditions transfer holds. Open questions include whether compact action codes cover more complex tasks, how different pretrained priors compare, and how the method performs on physical robots. Based on the supplied material, O²LaF’s central contribution is a latent-action framework that shares a pretrained prior while adapting separate models for inverse and forward dynamics.

Hugging Face Daily Papers

Comments

Checking sign-in status...

Loading comments...

Related articles