Mental World Modeling: World Models Need to Simulate Minds, Not Just Scenes

Introduction

World models are usually framed as internal simulators of the external environment: what objects exist, where they are, and how the visible scene will evolve. That view is useful for planning and action, but the paper Mental World Modeling argues that it leaves out a crucial part of human behavior. A model may describe the physical scene accurately and still predict the wrong human action, because people act based not only on what is physically present, but also on what they believe, want, intend, feel, and consider socially acceptable.

Key ideas

• Physical accuracy is not enough: Two scenes can look identical from the outside while leading to different decisions if the agents inside them hold different beliefs, memories, goals, emotions, or obligations.
• MWM treats mental variables as part of the world state: Instead of using mental-state explanations after the fact, Mental World Modeling makes beliefs, desires, intentions, feelings, and social constraints core components of the model.
• The model must render agent-specific observations: What a target person can see, hear, know, or infer may differ sharply from the full physical scene. MWM therefore asks the model to reconstruct the world from that person’s partial perspective.
• Actions update both the physical and mental worlds: An action can move objects, but it can also change what others know, expect, fear, or consider appropriate. Candidate actions must therefore be simulated as coupled physical-mental transitions.
• MENTIS provides an inspectable baseline: The authors instantiate the framework in MENTIS, a training-free system that decomposes prediction into state parsing, target-observation generation, action decomposition, coupled transition simulation, branch-level evaluation, and final decision selection.

What the experiments show

The work evaluates this idea on a manually constructed and quality-controlled dataset of situated decision scenarios spanning text, image, and sounding-video stories. Across experiments with eight modern LLM-based world models, the central finding is consistent: explicitly modeling mental state is necessary for robust human decision prediction.

This does not mean current LLMs have no ability to reason about beliefs or intentions. Rather, the paper highlights that such reasoning is often not maintained as a stable, structured state. By making the intermediate stages visible, MENTIS helps reveal where failures arise: the system may misread the scene, generate the wrong partial observation for the target person, update beliefs incorrectly, or evaluate candidate actions poorly.

Why it matters

MWM points to a broader evolution in world modeling: from simulating physical scenes to simulating the minds acting within those scenes. This shift matters for AI agents, embodied intelligence, human-AI collaboration, video understanding, and social reasoning. In real environments such as homes, classrooms, workplaces, or care settings, an AI system cannot rely only on an objective description of the world. It also needs to track how each person understands that world.

The paper’s deeper message is that behavioral prediction is not just a perception or chain-of-thought problem. It requires a persistent representation of mental context. A more capable world model may need to answer two questions at once: how the external world will change, and how the people inside it will interpret and respond to that change.

Source: Hugging Face Daily Papers