GE-Act 2.0 Explores Scalable Pretraining for World-Action Models

Introduction

A robot that sees an object still needs to reason about what the scene will look like after an action. World-action models (WAMs) address this problem by predicting future states that can guide control. They can potentially learn from both action-free video and action-labeled interaction, making them different from policies trained only on demonstrations with explicit action annotations.

GE-Act 2.0, introduced by the AgiBot Research Team, focuses on a less explored question: how should a world-action model itself be pretrained and scaled? Many existing systems inherit a pretrained video generator. That strategy provides a strong visual prior, but it leaves open whether the generative and action components can be learned directly from manipulation data and whether more data reliably translates into better control.

How the model is organized

GE-Act 2.0 initializes its trainable generative and action components from scratch. The architecture has three main parts:

• Control-oriented autoencoder (CoAE): It compresses observations while preserving information that matters for actions and instructions. The goal is not merely to reconstruct pixels, but to retain a compact representation useful for control.
• Single-step visual planner (SVP): It predicts a complete future state in one differentiable pass. This design allows visual planning and inverse dynamics to be pretrained separately rather than forcing the entire system into a long iterative generation process.
• Inverse dynamics model (IDM): Given a current state and a desired future state, it estimates the action needed to produce the transition.

The components are first trained on complementary data and then jointly optimized. The joint stage uses knowledge-aligned selective optimization, or KASO. Instead of treating every predicted future as valid supervision, KASO selects predictions that are behaviorally compatible with the recorded action. This is intended to reduce the mismatch between visual observations, action labels, and model-generated futures.

Evaluation and results

The paper evaluates pretrained checkpoints directly, without task-specific fine-tuning. The benchmark contains 100 tasks across 20 manipulation skill groups, with held-out scenes, backgrounds, lighting conditions, and object instances. This setup tests whether the model can transfer skills beyond the exact settings represented in training.

Scaling co-training data from 300 to 30,000 hours raises the success rate on G1-OP from 17.1% to 44.1%. On G2-90D, performance increases from 13.4% to 31.1%. The latter result is notable because G2-90D accounts for less than 2% of the co-training data, yet its inclusion is associated with a 17.7-point improvement. The authors interpret this as evidence of cross-embodiment transfer, although the result should not be read as proof that transfer will be equally strong for every robot platform.

Improvements appear in 19 of 20 skill groups for one setting and 18 of 20 for another. Skill-specific data coverage also correlates strongly with zero-shot out-of-distribution success, with reported Pearson and Spearman coefficients of 0.80 and 0.85. In language-grounded tests, the model identifies object, color, shape, and position references in at least 90% of trials. It also follows explicit instructions when they conflict with an already committed behavior or a conventional scene association.

Why it matters

The contribution of GE-Act 2.0 is broader than its individual modules. It frames world-action modeling as a pretraining problem in its own right, rather than as a visual generation capability borrowed from another domain. Separating state compression, visual prediction, and inverse dynamics gives different data sources clearer roles, while KASO offers a way to limit noisy or behaviorally inconsistent supervision.

The scaling results also point to an important qualification: more hours help, but coverage matters. Data must represent relevant skills, embodiments, and environmental variation. For robotic foundation models, increasing the raw volume alone may therefore be insufficient. GE-Act 2.0 provides evidence for a scalable route toward zero-shot manipulation, but robustness over long-horizon tasks, recovery from failures, and safety in physical deployment remain open questions.

Source: Hugging Face Daily Papers