Back to articles

Coda-DaVinci Uses 3D Proxies to Guide Video Generation

3 min read

Generating a video from text is only part of the challenge. A system must also control how objects move, preserve scene structure, and keep a sequence coherent as events unfold. Code2Video approaches this by having an agent build a 3D proxy in code and using that proxy to guide video generation. Coda-DaVinci investigates how to make this approach work better: how detailed should the proxy be, when should physics simulation be used, and how should prompts refer to the proxy?

What the study finds

  • Proxy detail involves a task-dependent trade-off. Coarse proxies tend to support more natural motion, while medium-grained proxies favor consistency. More geometry is therefore not automatically better; the useful level of detail depends on what the generation task demands.
  • Physics simulation and prompts aligned with the proxy both improve average quality, according to the study. Simulation supplies a motion reference, while proxy-grounded prompts connect language instructions to modeled structure and dynamics. The supplied material does not specify the size of these gains.
  • In long-video evaluations, the approach improves event structure and consistency across successive interactions. Transition stability and adherence to scene settings remain limitations.

Coda-DaVinci turns these observations into a task-adaptive framework. It can choose between proxy-guided and direct generation, and vary modeling granularity for different entities in a scene. It also coordinates simulation with prompts grounded in the proxy. A multi-agent harness preserves scene assets and supports repair: agents inspect outputs visually, identify issues, and use independent review to guide corrections. The design thus treats video generation as a sequence of linked modeling, generation, and checking steps rather than a single prompt-to-video operation.

The paper reports leading instruction-following performance on VTR and scene consistency on R2M. It also reports improved physical plausibility over direct generation and higher visual realism than the Code2Video baselines included in the comparison. These are reported evaluation outcomes, not a guarantee that the method will win on every task; the abstract provides no scores or detailed experimental breakdown. The long-video results likewise suggest progress without resolving transition reliability or scene-setting adherence.

There is also a data-quality caveat. A comment on the Hugging Face paper page says that the G1/PE1 changes in Table 2 do not match changes calculated from Table 1 averages, and notes a naturalness value falling from 1.968 to 1.950 in Table 1. The supplied material contains no author response or correction, so those table-level figures should be checked before being treated as settled. The broader contribution is a flexible control strategy: proxies can be adjusted to the task, while natural motion, consistency, and continuity remain competing goals.

Hugging Face Daily Papers

Comments

Checking sign-in status...

Loading comments...

Related articles