4DAnyone Reconstructs Dynamic Humans from Casual Monocular Video

Introduction

A compelling goal in 4D vision is to capture a person with an ordinary camera and obtain a dynamic 3D representation that can be viewed from new angles. A monocular video, however, exposes only one viewpoint. A reconstruction system must infer hidden surfaces while preserving identity, clothing, geometry, and motion across both space and time. 4DAnyone addresses this problem with a two-stage pipeline: generate multiview videos from an uncalibrated monocular input, then lift those videos into a 4D Gaussian Splatting (4DGS) representation.

The scaling problem

Camera-controlled video diffusion models can produce plausible images or videos from novel viewpoints. Plausibility alone is not enough for 4D reconstruction. A 4DGS pipeline may require tens of target views, while a diffusion transformer cannot process all of them in one forward pass. Splitting the views into groups creates two coupled problems:

• Reference context grows with every generated view. Conditioning on all previous views increases the reference context linearly. As the context expands, the model’s appearance guidance becomes less effective, making it easier to lose details such as clothing, texture, or personal identity.
• Target groups are isolated. Separately processed groups cannot directly coordinate their decisions. Small inconsistencies can accumulate into global drift in body proportions, surface shape, or other structural features.

4DAnyone introduces two complementary mechanisms. Reference Context Packing, or RCP, compresses the growing set of reference views into a fixed-length, mixed-resolution context. This keeps the reference-side complexity at O(1) with respect to the number of views while retaining appearance cues at multiple scales. Target Context Routing, or TCR, changes how target views are grouped during denoising. At high-noise stages, rotating group assignments allows information to travel between groups and supports global structural agreement. At low-noise stages, the process favors detail stabilization.

Once the multiview videos have been generated, they are used as the input for 4DGS reconstruction. In this design, the video diffusion model is not merely a tool for synthesizing attractive alternative views. It acts as an intermediate bridge that supplies the spatial evidence missing from a single camera stream.

Data and evaluation

The authors introduce MVGameHuman, a dataset created with an in-house game engine, and combine it with light-stage and in-the-wild video data for training. Experiments on DNA-Rendering and DyMVHumans reportedly show improvements over previous methods in both novel-view video quality and downstream 4DGS reconstruction, with robust generalization to in-the-wild footage. The supplied material does not include numerical scores, so the size of those gains cannot be independently quantified here.

Why it matters

The broader significance of 4DAnyone is its focus on consistency as a systems problem. RCP addresses the loss of useful reference guidance caused by an ever-growing context, while TCR addresses the global coordination problem created by grouped generation. Together, they target a practical mismatch between diffusion-model context limits and the large number of views demanded by dynamic reconstruction.

The approach could lower the capture barrier for digital humans, virtual try-on, game character production, film previs, and interactive telepresence. It does not remove the inherent ambiguity of monocular vision, however. Occlusion, unknown depth, fast motion, and generative hallucination remain important concerns. 4DAnyone is therefore best viewed as a generative route toward more accessible 4D human reconstruction, not as a complete solution to every uncertainty in monocular capture.

Source: Hugging Face Daily Papers