PAWBench Asks Whether Video Models Can Model Probabilities, Not Just Possibilities

Introduction

Video generation systems are increasingly described as world models. Yet producing a coherent and physically plausible clip is only one part of what a world model should do. Many real processes are inherently ambiguous: given the same initial observation and the same action, more than one outcome may be valid. A useful model should therefore represent not only one plausible trajectory, but the range of futures and the likelihood of each one.

PAWBench, introduced in the paper, focuses on this often-missed requirement. The authors call it probabilistic alignment: conditioned on the same observation and action, a model should generate a distribution of behaviors that agrees with the reference distribution.

Why a single good video is not enough

Most existing evaluations emphasize individual-video plausibility. They ask whether a generated sequence looks realistic, follows the prompt, or maintains reasonable temporal and physical consistency. Those tests can be useful, but they do not reveal whether a model has learned the uncertainty of the environment.

A system could repeatedly choose one visually convincing outcome while ignoring other outcomes that are also likely. Conversely, it might produce a wide variety of clips, but assign the wrong frequency to each behavior. Both cases expose a weakness that single-sample evaluation can miss.

PAWBench addresses the issue through three related ideas:

• Repeated rollouts: the same initial condition and action are sampled multiple times.
• Outcome-level analysis: videos are converted into empirical distributions over observable physical behaviors.
• Joint evaluation of coverage and probability: the model must recover both the range of valid outcomes and their relative likelihoods.

The accompanying PAWEval protocol is designed to make this process systematic. Instead of treating a video generator only as a tool for producing attractive clips, it evaluates the system as a stochastic sampler of world dynamics.

What the benchmark finds

The study evaluates 11 current systems across 50 scenarios. Its central finding is a substantial gap between visual plausibility and probabilistic alignment. No model consistently recovers the reference probabilities while also covering the full set of valid behaviors.

This result does not mean that every generated clip is physically implausible. Rather, it shows that local realism and distributional correctness are different capabilities. A model may produce a convincing sample yet fail when asked to represent uncertainty across repeated generations. That distinction matters whenever a downstream agent must plan under multiple possible futures.

The authors also explore whether three factors can reshape a model’s predictive distribution: language prompts, the sampling of initial noise, and model training. These experiments frame randomness as more than a source of artifacts. If controlled appropriately, it may become part of how a generator expresses alternative futures. The supplied material, however, does not claim that any of these factors has already solved the alignment problem.

Why it matters

For robotics, interactive simulation, and planning agents, selecting one plausible future is not enough. A decision system needs to estimate which outcomes are possible and how likely they are. Missing a lower-frequency branch can be especially consequential when that branch affects safety or long-horizon behavior.

PAWBench’s broader contribution is therefore methodological. It shifts the conversation from “Does this video look right?” to “Does the model reproduce the right distribution when the experiment is repeated?” That change encourages future benchmarks and training methods to treat uncertainty as a first-class target.

The benchmark does not present current video generators as complete world models. Instead, it identifies a concrete limitation in the field’s current framing: generating realistic-looking futures and sampling the world’s possible futures with the right probabilities are not the same achievement.

Source: Hugging Face Daily Papers