JoyAI-Echo-1.5 Pushes Audio-Visual Generation Toward Persistent Stories and Interactive Worlds

Introduction

Video generation is moving beyond isolated clips. The next challenge is to create stories that can continue over many shots and worlds that respond coherently to user input. This requires more than visual quality at a single time step. Characters must retain their appearance, voices must remain identifiable, camera motion must follow controls, and the generated environment must remain stable as the rollout becomes longer. JoyAI-Echo-1.5 addresses these requirements with a unified audio-visual generation framework.

Two purpose-built variants

The paper presents two complementary versions:

• Long-video generation: Its composable cross-shot memory aggregates visual evidence from multiple earlier shots. Instead of relying only on the latest frame or one reference image, the model can reuse information distributed across the narrative history. This is intended to improve the persistence of character appearance across flexible combinations of text, image, and memory conditions.
• Interactive world modeling: The second variant converts heterogeneous navigation inputs into calibrated metric 6-DoF camera trajectories. These trajectories are injected through a geometry-aware conditioning pathway, allowing the model to support different controllers and viewpoints without tying interaction to one input format.

The audio component is equally important. Speaker cues are derived from speech-filtered full-shot audio, giving the model a signal for maintaining voice identity across a scene rather than treating sound as an isolated accompaniment. Together, these mechanisms frame identity as a cross-modal and cross-time problem.

Training for long rollouts

Long-horizon generation also exposes a training mismatch. During training, a model can receive clean historical context, while at inference it must condition on its own imperfect outputs. JoyAI-Echo-1.5 transforms a bidirectional audio-visual backbone into a causal few-step generator using progressive teacher forcing. It further applies short- and long-horizon Self-Gradient Forcing on self-generated rollouts, encouraging the model to learn from the conditions it will actually encounter during deployment.

This design does not eliminate drift automatically, but it directly targets one of the main causes of drift: the gap between ground-truth histories and model-generated histories. The combination of memory and rollout-oriented training is therefore more consequential than either component in isolation.

Results and broader significance

The abstract reports improvements over existing long-video baselines in cross-shot consistency, visual quality, text alignment, and speech fidelity. For the world-model variant, the paper reports an average score of 81.7 on WBench, along with leading visual quality and long-horizon persistence on SANA-WM-Bench. The supplied material does not include the full evaluation protocol or comparison tables, so these results should be interpreted as the paper’s reported claims rather than independently verified conclusions.

The broader contribution is a shared view of persistent generation: a system must remember who is present, understand how the user wants to move through a scene, and remain stable as time expands. If the approach scales, generative systems could evolve from producing disposable clips into maintaining story spaces or explorable visual environments. Important open questions remain, including failure boundaries over even longer rollouts, geometric accuracy under complex controls, and whether identity memories remain reliable when scenes, viewpoints, or speakers change substantially.

Source: Hugging Face Daily Papers