Apple-PI: A New Benchmark for Physical Reasoning in Video Models

Video generation models are often described as emerging world models. But a plausible clip does not prove that the model understood the underlying physics. Apple-PI is designed to close that gap by evaluating whether video models arrive at their outputs through faithful, law-grounded reasoning.

What Apple-PI introduces

The benchmark has three parts:

• Orchard, a dataset of 400 videos spanning 10 canonical classical mechanics tasks.
• A three-stage protocol based on scientific reasoning: Perception, Formulation, and Deduction.
• A hybrid evaluation suite that combines MLLM-based subjective scoring with physics-law-grounded objective measures.

A notable design choice is chain-of-frames prompting on infographic-annotated first frames. The generated video is treated as the model’s visible reasoning trace, which allows the benchmark to pinpoint where the model breaks down, not just whether it fails.

Why this matters

Apple-PI shifts evaluation from output-only realism to process-aware diagnosis. It also separates single-law tasks from multi-law tasks, which helps distinguish confounder-free diagnosis from generalization testing. That makes the benchmark more useful for understanding what video models actually learn about the physical world.

Testing 11 models shows that today’s systems are still far from dependable law-grounded world simulators. The best video model reaches only 0.473. The paper’s stage-, pillar-, and source-resolved analyses further reveal three recurring issues:

1. a bottleneck from Perception to Formulation to Deduction,
2. weak multi-law state transfer,
3. a persistent Sim-to-Real gap.

Broader impact

The main contribution of Apple-PI is diagnostic clarity. It helps researchers see whether a model misunderstands the scene, fails to encode the correct law, or cannot carry that law through to the final prediction. That kind of analysis is important for future video models, world models, and embodied AI systems.

If video models are to become trustworthy physical simulators, evaluation must move beyond “does it look right?” and toward “did it reason in a law-consistent way?” Apple-PI offers one of the first concrete steps in that direction.

Source: Hugging Face Daily Papers