WorldSculpt Builds Compositional 3D Worlds from Grounded Video

Introduction

Turning a real-world video into an editable and interactive 3D world requires more than recovering visible surfaces. Objects may be seen from only a few angles, large portions can be hidden by neighboring objects, and the final representation must preserve object-level structure rather than collapse everything into one static surface. The paper WorldSculpt: Generating Compositional Worlds from Grounded Videos addresses this problem with a generative, compositional formulation.

Instead of reconstructing a cluttered environment as a single mesh or scene representation, the method aims to produce a collection of individual object meshes placed in a shared world coordinate frame. This format is particularly relevant to applications that need to move, edit, simulate, or reason about scene objects independently.

Core approach

Geometry-first reconstruction methods can remain faithful to observed pixels, but they often produce incomplete geometry when objects are severely occluded. Existing compositional approaches that use generative priors have generally focused on simpler scenes and have not demonstrated the same scale of object-level decomposition.

WorldSculpt takes a different route: it adapts a strong single-object 3D generative prior to multi-view observations. The implementation is based on Pixal3D, which is extended with a conditioning pathway for multiple posed views. These observations provide evidence about an object’s appearance, location, and visible structure, while the generative prior supplies a way to infer a more complete mesh from partial evidence.

A notable aspect of the design is the gap between training and deployment. The model is fine-tuned entirely on single objects represented in canonical space, rather than on complete crowded scenes. According to the paper, this prior can still generalize to large scenes with severe occlusion without scene-level training. The result is a modular pipeline in which object generation and world-level placement are combined after the individual instances are grounded in observations.

Benchmark and evaluation

The authors introduce UE-MeshyScene, a photorealistic benchmark for densely cluttered environments. It contains scenes with hundreds of objects, per-object annotations, and ground-truth meshes, allowing evaluation beyond image-level similarity and toward object-aware 3D reconstruction.

The method is evaluated in single-object, controlled multi-object, and UE-MeshyScene settings. The paper reports consistent improvements over prior approaches, with larger gains as scene complexity and occlusion increase. It also demonstrates a broader conversion scenario: generated 3D Gaussian Splatting worlds, including Marble and HY-World 2.0, can be converted into compositional mesh scenes.

Why it matters

The main contribution is a useful division of labor. Video observations anchor the reconstruction in a particular world, the single-object prior helps complete partially hidden geometry, and the shared coordinate frame turns separate predictions into a coherent scene. This could support game asset creation, AR/VR, simulation, and robotics, where an editable collection of meshes is more useful than an inseparable reconstruction.

The available material does not establish that the approach solves general-purpose world reconstruction. Questions remain around distinguishing similar instances, recovering fine structures, preserving material appearance, and resolving conflicts between generated geometry and limited visual evidence. WorldSculpt is therefore best understood as evidence that single-object priors can scale toward compositional worlds, not as a final answer to every cluttered-scene reconstruction problem.

Hugging Face Daily Papers