Pelican-Sim 1.0 Builds a General Action-Conditioned World Model for Robots

Introduction

World models are becoming an important component of embodied AI because they can forecast what an agent will observe after taking an action. Such forecasts allow a robot to test decisions, generate training data, and evaluate policies without relying exclusively on costly real-world interaction. Pelican-Sim 1.0 targets this role as a general simulator that can work across robot embodiments, scenes, and tasks.

Four design choices

The first design choice is a unified action representation. Pelican-Sim maps control signals from mainstream robot embodiments into a 28-dimensional action-value space. One model can therefore be applied across heterogeneous devices instead of requiring a separate action interface and model for every robot. The challenge is that different mechanisms and control conventions still produce different dynamics, which the simulator must learn to distinguish.

The second is action-visual injection. Rather than simply concatenating an action vector with visual features, the system uses URDF and camera rendering to create action-conditioned videos. These rendered sequences provide a visual bridge between a command and its expected pixel-level consequences. According to the supplied material, this approach improves PSNR by 0.904 over alternative fusion baselines.

Pelican-Sim also uses sparse mixture-of-experts layers. Robot bodies, environments, and tasks can exhibit substantially different dynamics, so routing computation through selected experts gives the model additional capacity for heterogeneous behavior. The MoE layers also absorb the action modality and reduce conflicts between visual and control information. Compared with a dense backbone, the design lowers FVD by 6.530.

The final component is efficient rollout generation. Causal adaptation and few-step distillation turn the original 35-step generation process into a four-step autoregressive simulator, producing a reported 5.67-fold speedup. Faster rollouts matter because planning, policy search, and synthetic-data generation may require many simulated trials.

Results and applications

The model is trained on approximately one million real-world and simulated trajectories. Relative to the strongest evaluated baselines, PSNR improves by 4.636 on AgiBotWorld Beta, 2.080 on RoboMIND, and 10.343 on RoboTwin. On RoboTwin, the adapted EWMBench DYN score increases by 0.426.

The reported gains also extend beyond visual prediction. In RoboTwin experiments, adding 500 generated trajectories to 50 demonstrations per task raises policy success from 70% to 93%. The simulator reaches a Pearson correlation of 0.994 for policy evaluation across five checkpoints. Relative success gains are 47.7% for action selection and 20.3% for policy improvement.

Why it matters

Pelican-Sim frames a world model not merely as a video predictor, but as an action-aware simulator intended to support robot decisions. Its unified action interface could make data and models more reusable across embodiments, while action-rendered videos offer a practical way to align symbolic control with visual consequences. The four-step rollout further improves its usefulness for large-scale planning and data generation.

The supplied material does not establish how the system behaves during long-horizon planning, under distribution shifts in real deployment, or with previously unseen robot structures. More closed-loop physical experiments will be needed to determine whether improved prediction consistently translates into reliable control.

Source: Hugging Face Daily Papers