StreamEval: Real-Time Video Generation Still Struggles to Follow Instructions

Introduction

Video generation is moving beyond the creation of isolated clips. The next generation of systems is expected to keep producing content while responding to requests that arrive during playback. A user may ask an avatar to change its performance, alter a generated environment, or edit an existing visual reference without losing the identity and elements already established. In this setting, visual realism is only one part of the problem. The system must also understand when a request arrives, apply the intended state change, and preserve everything that should remain stable.

StreamEval addresses this gap with a unified benchmark for interactive streaming video generation. Rather than judging a finished clip alone, it tests whether a model can manage a continuing stream under timed, multimodal interaction requirements.

Key points

• Five task families. The benchmark covers avatar dialogue, avatar performance, world generation, instruction editing, and reference editing. Together, these tasks represent both character-centric and scene-centric interaction.
• Explicit temporal constraints. StreamEval includes 300 scripted test cases lasting between 15 and 300 seconds. Each case defines initial assets, incrementally arriving requests, required state changes, and content that must be preserved.
• Four evaluation dimensions. Models are assessed on task execution, state consistency, visual fidelity, and audiovisual coordination, using ten task-specific metrics scored from 0 to 100.
• Following requests remains difficult. Among seven avatar-performance models, the highest Following score was 36.90. For text-conditioned world generation, the highest Following score was 25.93. Among three reported reference-editing models, the best Edit Success score was 15.00. The gap indicates that attractive output does not necessarily mean successful interaction.

Why the benchmark matters

StreamEval makes several requirements of streaming systems measurable. Conventional video-generation evaluations often emphasize appearance, motion quality, or identity retention in a single clip. An interactive system must solve a broader problem: it has to maintain a coherent state across time, determine whether a new instruction changes or extends that state, and execute the request without damaging protected content.

This distinction is important because a model can look impressive at the frame or short-clip level while failing over a longer interaction. A missed request, a delayed change, or an unintended alteration to a character can undermine the entire experience even when the resulting frames remain visually polished. StreamEval therefore treats execution and consistency as first-class capabilities rather than side effects of generation quality.

For model developers, the results point toward priorities beyond resolution and appearance. State representations, request scheduling, temporal memory, and verifiable edit outcomes all become central engineering concerns. For users, the practical question is not simply whether a system can make a beautiful video, but whether it can keep that video coherent while responding accurately to the next instruction.

The evaluation covers 37 distinct models and offers a common basis for comparing future systems. As interactive video is applied to avatars, simulated worlds, and creative tools, reliable execution under timing and preservation constraints may become one of the clearest indicators of product maturity.

Source: Hugging Face Daily Papers