Lab2Sim Turns Experiment Videos into Interactive Simulations

Introduction

A science experiment video can show how equipment is arranged, which actions are performed, and how the setup changes over time. Yet watching a demonstration is not the same as conducting an experiment. Learners cannot try an invalid action, observe why it fails, or explore how one operation depends on a previous state. Lab2Sim addresses this gap by studying how real-world experiment videos can be converted into interactive virtual experiments.

Key points

• A benchmark for video-to-experiment conversion. Lab2Sim contains 218 science experiment videos, together with evidence-grounded specifications and 11,881 rubric items. These annotations are intended to measure whether a generated simulation follows the experiment shown in the source video.
• Runnable does not mean faithful. Existing multimodal models can often produce pages that load and respond to user input. Their larger weakness is reconstructing the apparatus itself: the relevant objects, their arrangement, and the relationships that make an operation meaningful.
• Procedures are represented hierarchically. EduVideo2Code extracts experimental steps at multiple levels and encodes them as explicit state machines. This gives the generation process a representation of both the next action and the conditions under which that action is allowed.
• The output is checked by execution. Instead of accepting the first generated page, the system runs it, examines the resulting interactions and state transitions, and uses that evidence to refine the code.
• A notable fidelity improvement. Evaluations with two backbone models show gains across all four reported dimensions, with apparatus fidelity increasing by 24.15 points.

Why it matters

Lab2Sim frames virtual-lab generation as more than a visual imitation or a front-end coding exercise. A beaker, stand, tube, or other piece of equipment is not merely decoration. Its placement and connection can determine which actions are valid and which state changes should occur. The state-machine representation provides a useful bridge between visual observations and executable interaction logic.

The work also highlights an important evaluation principle for educational AI. A page that renders successfully may still teach the wrong procedure if its apparatus is inaccurate or if users can perform operations in an impossible order. By combining evidence-grounded scoring, structured procedure extraction, and execution-based refinement, Lab2Sim offers a more demanding way to assess video-driven educational content generation.

The results do not eliminate the broader challenge. The reported gains show that structure and verification help, while the baseline difficulty in reconstructing apparatus indicates that multimodal systems still struggle with detailed physical relationships. Future virtual experiments will therefore need both stronger visual understanding and reliable mechanisms for enforcing experimental constraints.

Source: Hugging Face Daily Papers