PrimeSim Builds a Long-Horizon Robot Simulator from Cross-Embodiment Data

Introduction

A useful robot world model must do more than produce plausible-looking video. It needs to respond to commanded actions, preserve the same scene across camera views, and remain coherent when its own predictions are fed back over many steps. PrimeSim is designed around these requirements. Starting from a pretrained video generator, the system is adapted with large-scale robot demonstrations and then posttrained as an interactive, action-conditioned simulator.

How PrimeSim is built

The first stage is progressive midtraining on 30,110 hours of demonstrations collected across 22 robot platforms. The data spans different embodiments and control interfaces, creating a representation challenge: an action that is meaningful for one robot may not have a direct equivalent on another. PrimeSim addresses this with a unified cross-embodiment action representation. A frame-aligned dual-path conditioning design further connects action inputs to the visual changes they are expected to cause.

The system also treats multiview generation as a consistency problem rather than as independent video synthesis. Its view-aware interleaved attention mechanism includes explicit camera identities. Features from different cameras can exchange information about shared objects and scene state, while view-specific representations are retained. In principle, this reduces contradictions in object location, contact, and deformation across views.

The second stage converts the original bidirectional video generator into a more efficient autoregressive simulator. PrimeSim’s “fast video, slow memory” framework combines causal generation, self-rollout training, and hierarchical temporal memory. Instead of learning only from real histories, the model is trained on histories that it generated itself. This exposes it to the distribution shift that occurs at inference time, when small prediction errors can otherwise compound over successive chunks. A selected KV-cache memory carries longer-range context between those chunks without requiring the entire history to be recomputed.

Results and limitations

PrimeSim runs at 10.1 FPS while generating three camera views. The supplied results report the best overall score on EWMBench and leadership on four of five WorldSimProbe suites among the compared open-source action-conditioned world models. The model still trails the field on fine-grained local calibration, so visual plausibility and broad rollout stability should not be confused with exact physical prediction.

The system also adapts to held-out robot platforms within 100 to 200 gradient steps, whereas the pretrained backbone does not show the same transfer behavior. In a reinforcement-learning experiment, GRPO post-training inside PrimeSim raises LIBERO success from 61.7% to 73.2%. Its success predictions are reported to rank policies similarly to a physics simulator, suggesting a possible role as a cheaper policy-screening environment.

All experiments, however, were conducted in simulation, including AgiBot G2, LIBERO, and RoboTwin settings. No physical robot was used, and closed-loop hardware deployment remains future work. The reported speed therefore demonstrates an inference capability, not yet a validated real-world control loop.

Why it matters

PrimeSim’s broader contribution is to combine action grounding, multiview modeling, memory, and policy evaluation in one simulator-oriented pipeline. If the approach transfers reliably to physical platforms, it could reduce the amount of robot time required for policy search and make cross-embodiment training more practical.

The open questions are substantial: long-horizon coherence does not guarantee accurate local physics, and simulation-to-real transfer may expose failures not visible in the current benchmarks. PrimeSim is best understood as a step toward generalist robot world models, rather than a finished replacement for physical experimentation.

Weights, inference code, and training code are planned for release on November 6.

Source: Hugging Face Daily Papers