PPMem Gives Video World Models a Persistent Planar Memory

Introduction

A video world model needs more than short-term visual continuity if it is expected to explore an environment. As the camera moves, the model must remember where surfaces are, how rooms connect, and which structures have already been observed. Without such a state, long rollouts can gradually lose their spatial identity: walls may shift, layouts may drift, and newly generated views may no longer agree with earlier ones.

PPMem, presented by Robbyant Research, addresses this problem with a persistent memory built from plane primitives. Rather than storing the scene as a large collection of image-aligned points, it focuses on dominant structural surfaces such as walls, floors, and facades.

How it works

PPMem extends a video diffusion Transformer with a planar prediction head and a Depth Token Fusion (DTF) module. During autoregressive generation, every video chunk predicts RGB frames and planar geometry together. The newly inferred planes are then merged with the existing memory. For later chunks, that memory is rendered into depth and injected through DTF, giving the generator an explicit geometric reference for the scene it has already built.

• A structural representation: A plane is a natural abstraction for large, visually important surfaces. Each primitive is described with seven parameters, making the representation compact while retaining orientation and spatial meaning.
• Geometry as conditioning: The persistent planes are not merely an output for evaluation. Their rendered depth is fed back into generation, linking future frames to a maintained scene state.
• Progressive refinement: The memory can be updated as new viewpoints reveal more of the environment. The merging procedure allows existing geometry to be refined rather than treating the first prediction as final.
• Multiple geometric outputs: Because RGB and geometry are predicted by a shared backbone, the system also produces metric-scale depth, surface normals, and surfaces that can be exported for downstream use.

According to the paper, a full-scene memory can contain roughly 100,000 plane primitives and occupy about 5MB. The authors describe this as three orders of magnitude more compact than per-pixel point-cloud representations. The significance is not only the smaller footprint: planes provide an explicit, interpretable description of scene structure instead of an unorganized set of samples.

Results and implications

PPMem is evaluated on DL3DV and RealEstate10K. The paper reports better video quality than earlier 3D-grounded world models, while jointly producing depth, normals, and planar geometry. It also reports maintaining coherence for hundreds of frames under substantial viewpoint changes.

The broader contribution is a different view of memory in world models. Memory is treated as a persistent geometric state that can be queried and revised, rather than just a longer visual context window. This could be useful for navigation, embodied agents, and interactive environment generation, where a model must repeatedly revisit parts of a scene.

There are also clear boundaries to the abstraction. Planes are well suited to architectural and other piecewise-planar structures, but curved surfaces, small objects, and dynamic elements may require complementary representations. How such memories behave in more cluttered, changing environments remains an important direction for future work.

Source: Hugging Face Daily Papers