VideoCoCo Turns Blender Code into Process-Level Reasoning for Video Generation

Introduction

Text-to-video systems have made impressive progress in visual fidelity, but realistic motion remains a hard problem. A short prompt can describe what should appear in a video, yet it often leaves the actual temporal process underspecified: how objects move, collide, accelerate, or interact over time. VideoCoCo addresses this gap by shifting part of the reasoning burden from the generative model to executable simulation.

The paper frames this idea as Code-as-CoT. Instead of using chain-of-thought as a textual explanation or a sparse plan, VideoCoCo uses Blender code as a process-level reasoning artifact. Given a text prompt, an agent writes a Blender program that explicitly defines the scene and its temporal evolution. The program is then executed to produce a deterministic spatiotemporal draft, which is later converted into a photorealistic video by a draft-conditioned video editor.

Key Points

• A dual-engine architecture: VideoCoCo separates process reasoning from visual rendering. The simulation side handles controllable dynamics, while the generative video engine focuses on high-fidelity appearance.
• Executable intermediate representation: Blender code is not merely a description. It can be run, inspected, adjusted, and reused, giving the system a more concrete handle on motion and scene evolution.
• Deterministic spatiotemporal drafts: By executing the generated program, the framework obtains a structured draft that encodes the intended dynamics across time, reducing the need for the video model to infer everything implicitly.
• Dataset for draft-conditioned editing: The authors construct VideoCoCo-3K, a curated collection of draft-instruction-target triplets, to help adapt the video editor to simulated inputs.
• Reported benchmark improvements: According to the paper, VideoCoCo improves the OmniWeaving baseline from 0.475 to 0.558 on PhyGenBench and from 52.18 to 77.88 on VBench-2.0, achieving the best average score on both benchmarks.

Why It Matters

The significance of VideoCoCo lies in its decomposition of video generation. Rather than asking one model to infer both the physics and the pixels from a compressed prompt, the framework first builds a procedural version of the world and then lets a generative model refine its appearance. This makes the process more controllable and more inspectable than purely implicit generation.

The approach also reflects a broader trend in video AI: combining generative models with simulation, programmatic world construction, and agentic planning. There are still open questions, including how reliably agents can write correct Blender code, how far simulated drafts can cover real-world complexity, and how faithfully the video editor preserves the intended dynamics. Even so, VideoCoCo offers a compelling design pattern: generate the process first, then generate the pixels.

Source: Hugging Face Daily Papers