From Pixels to States: Why Interactive World Models Are Not Game Engines Yet

Introduction

Video generation models have made the idea of an “AI game engine” feel less speculative. If a model can take player inputs and predict what the next frames should look like, it may appear to offer a data-driven alternative to hand-built rendering and simulation pipelines. But the paper From Pixels to States: Rethinking Interactive World Models as Game Engines argues that this view is incomplete.

A real game is not merely a stream of plausible images. It is a loop: the player acts, the game state changes according to rules, and the resulting state is rendered back to the player as an observation. By using this action-state-observation loop as its organizing lens, the paper asks what current interactive world models still lack before they can behave like genuine game engines.

Key points

• Action control is more than conditioning. Feeding keyboard, mouse, or controller inputs into a video model is only the first step. A jump, dodge, attack, or movement command must produce outcomes that depend on the current game conditions. The same action should behave differently when stamina, collision, enemy position, or character status changes.

• Game state dynamics remain the hard problem. The paper emphasizes that interactive worlds require evolving internal states: health, inventory, position, enemy behavior, quest progress, and other variables must be tracked over time. Pixel-level prediction can look convincing in the short term while failing to preserve the underlying rules over longer horizons.

• Persistence matters. Consequences in games are not disposable visual effects. A defeated enemy should not simply reappear, an opened door should remain open unless the rules say otherwise, and consumed resources should be remembered. The authors therefore treat state-observation persistence as a distinct capability rather than a side effect of video quality.

• Real-time generation is a fundamental requirement. Game engines must respond immediately within an interactive loop. Even a visually impressive model is not enough if latency is high or inference speed is unstable. For generative world models, real-time operation is not just an engineering convenience; it is part of the definition of playability.

A data-oriented contribution

Beyond its conceptual framework, the paper introduces a scalable data engine built around Black Myth: Wukong. According to the abstract, the system collects more than 90 hours of gameplay with frame-aligned player actions, ground-truth game states, visual observations, and structured as well as semantic annotations.

This is notable because most video-centric datasets mainly expose what appears on screen. A state-aware game world model needs more: it must learn how actions change hidden variables and how those variables later shape visual outcomes. Aligning actions, states, and observations may therefore become a key ingredient for training models that behave less like video predictors and more like interactive simulators.

Why it matters

The paper’s importance lies less in proposing a single new model and more in clarifying the criteria for the next generation of AI-driven interactive worlds. It shifts the discussion from “Can the model generate realistic frames?” to “Can the world obey rules, remember consequences, and respond in real time?”

For game development, this suggests that generative models may first become tools for prototyping, visual preproduction, content creation, or assisted simulation rather than immediate replacements for traditional engines. For AI research, games remain a demanding testbed where vision, control, memory, rule-following, and latency all collide. The next stage of world-model research may be defined not by sharper pixels, but by more reliable states.

Source: arXiv