InfinityEdit Brings Instruction-Based Editing to Infinite Video Streams

Introduction

Many current video-editing systems assume that the source is a completed clip with a fixed duration. Their outputs are aligned with the source frame by frame, making this setup suitable for changes such as restyling or modifying content within an existing time span. It is much less suitable for a live game, an ongoing shot, or a camera movement that should begin now and continue into frames that have not yet been generated.

InfinityEdit studies this open-ended setting under the name “infinite video editing.” Given a preceding segment and an edit request, the system must generate the next segment so that it both continues the stream and reflects the requested change. New instructions can arrive later, turning editing into a repeated streaming process rather than a one-time transformation of a finished file.

Key points

• The task is future-frame generation. The model is not merely rewriting known frames. It must preserve temporal continuation while applying the requested edit, creating a tension between faithfulness to the stream and responsiveness to instructions.
• A lightweight adapter adds edit control. InfinityEdit uses three attention components. History cross-attention guides denoising with input frames, temporal causal self-attention allows temporal information to move from earlier frames to later ones, and edit cross-attention injects the instruction into generation.
• Editing is activated locally. During inference, the adapter is used in the chunk where an edit request arrives. Subsequent chunks are generated by the original streaming model with a reset anchor frame, a design intended to preserve the model’s existing ability to generate indefinitely.
• Data construction supports the setting. The authors first develop a data-collection pipeline for infinite video editing and then train the adapter on the collected data, focusing on stability as edits accumulate.

Why it matters

The main contribution is a shift in the temporal scope of video editing. In live content, interactive games, and continuously generated scenes, users may not have the complete video in advance. They may also want an instruction to take effect from a particular moment onward without reprocessing everything that came before. A stable streaming editor could therefore treat instructions as controls that enter the generation process over time.

The approach also illustrates a practical integration strategy: add a relatively small editing layer while allowing the pretrained streaming generator to handle later chunks. This may reduce interference with the base model’s continuation behavior, although the supplied material does not establish production readiness or provide a full set of quantitative results.

Important open questions remain, including how conflicting instructions are handled, how identity and scene consistency behave over very long sequences, and how quality changes under more complex edit schedules. The paper reports that InfinityEdit faithfully continues streams and remains stable across unbounded edit sequences in its experiments, but the available summary does not include detailed metrics. Its clearest significance is therefore architectural: it treats video editing as an ongoing interaction with a generator rather than a fixed-clip rewrite.

Source: Hugging Face Daily Papers