From Generation to Simulation: How Close Are World Models to True Simulators?

Introduction

World models are increasingly being presented as environments in which intelligent agents can act, not merely as systems that generate plausible video. As diffusion models and large-scale video generation improve, comparisons with physics engines, game engines, and reinforcement-learning environments have become increasingly common. Yet visual plausibility is not the same as simulation. A simulator must expose a controllable and repeatable world whose state can be computed, queried, and used for decision-making.

The study From Generation to Simulation: How Far Are World Models from Being True Simulators? evaluates that gap from the perspective of simulation rather than generation.

Key findings

The authors use eight capabilities of traditional simulators as an external yardstick: asset construction, physics engines, interaction, controllability, stability, state feedback, diversity, and evaluation metrics. They then organize 200 representative papers published between 2018 and June 2026 across three major routes: latent dynamics, video generation, and joint-embedding prediction.

The result is more nuanced than a simple verdict that world models are not ready. In selected settings, they already support meaningful interaction and action-conditioned control. This gives them a form of functional substitution for narrowly defined tasks. However, that substitution is usually bounded by a particular dataset, task, or visual scenario rather than backed by the general guarantees expected from a simulator.

Physics is one central gap. A model may produce a visually coherent outcome over a short horizon without representing physical rules in a way that is explicit, verifiable, or reliably extrapolatable. Long-horizon stability is another problem: small prediction errors can accumulate during rollout, making future states less consistent and less reproducible.

The survey identifies state feedback as the most neglected weakness across technical routes. Conventional simulators typically let programs query positions, velocities, or physical parameters so that controllers can make decisions from structured state. Many world models instead expose primarily images or videos. Among 163 papers with implementations, only six provide a runtime interface for querying entity states or physical parameters. In other words, a model can appear interactive while remaining difficult to use as a rigorous environment for robot learning or policy evaluation.

Why it matters

The study argues that visual quality, realism, and rollout length are insufficient evaluation criteria. For embodied AI and reinforcement learning, researchers also need to know whether actions have a unified meaning, states can be read directly, outcomes can be reproduced, and predictions improve downstream policies. Clear physical constraints matter as well.

The authors propose six directions: formalized physics, a unified action interface, first-class state feedback, improved long-horizon stability, evaluation based on downstream utility, and hybridization across world-model routes. The next phase of progress may therefore be defined less by generating more convincing scenes and more by building worlds that agents can observe, control, verify, and simulate over time.

Source: Hugging Face Daily Papers