VideoMemBench Tests Whether Video Models Really Remember

Introduction

A multimodal large language model may be able to process an hour-long video, but that does not mean it can reliably remember what happened earlier in a conversation. In practical use, a user may first ask about one scene, then ask how it relates to another event. They may also ask the model to make a judgment and later build on that judgment. The system must therefore retrieve information from the video while preserving the conversational state it has already established.

VideoMemBench is designed to measure this ability, which the paper calls in-context video memory. The benchmark asks whether a model can reuse both what it saw in the video and what it said in earlier turns.

What the benchmark changes

• From isolated questions to linked dialogue. Conventional video QA questions are usually independent and answerable directly from the video. They do not force a model to retain earlier turns. VideoMemBench connects later questions to information introduced before them.
• Two types of dependency. An episodic dependency points back to a visual episode described in an earlier turn. It tests memory of the video. An answer dependency builds on the model’s own earlier response. It tests memory of the conversation and the model’s stated position.
• Controls against superficial shortcuts. Each dependency must be necessary: the later question cannot be answered without the earlier turns. It must also avoid leakage, meaning the later wording cannot simply restate the information it supposedly requires. A planner, generator, and judger help construct the dialogues, followed by human expert review.
• Long-range links. The dataset includes multi-parent dependencies and long-range callbacks, making it more demanding than a sequence of unrelated video questions.

Results and implications

VideoMemBench contains 326 videos and 2,476 dialogue turns. Of these, 1,387 depend on earlier turns. The authors evaluate more than 20 open- and closed-source multimodal models. Claude Opus 5 performs best, but its overall score is only 45.7%. The result suggests that processing long videos and maintaining useful memory across a dialogue remain distinct capabilities.

The challenge is not simply fitting more frames or tokens into a context window. A model must decide which historical details matter, retrieve the right visual episode, distinguish evidence from its own previous claims, and preserve consistency as the conversation develops. Longer context alone may therefore be insufficient without better state tracking and history retrieval.

The benchmark is useful because it connects video understanding with conversational memory in a controlled setting. It can support evaluation of video assistants, long-running agents, meeting analysis systems, and educational tools. Its necessity and non-leakage criteria also address a common weakness in benchmark construction: questions that appear to rely on history but can actually be solved from the video alone.

For a video assistant to participate in review, instruction, or event analysis over time, it must remember more than visual content. It must also remember what the conversation has already established. VideoMemBench offers a concrete starting point for testing that requirement.

Source: Hugging Face Daily Papers