Kaiming He’s Team Introduces VISTA for Visual Memory in Multimodal Agents

Introduction

A multimodal model may be able to interpret an image, but that does not mean it can preserve and reuse visual experience over a long interactive task. In the paper “VISTA: A Visual Harness for Reasoning in an Interactive World,” Kaiming He’s team proposes a visual-native harness aimed at this memory problem.

VISTA does not retrain the underlying model. Instead, it changes how the model receives, stores, and retrieves visual evidence. Raw observations from the environment are kept outside the active context and brought back only when they become relevant. This avoids losing early scenes to context limits or replacing important details with incomplete textual summaries.

How VISTA works

• Visual observation: The current environment is shown directly to the multimodal model. In the ARC-AGI-3 experiments, the official 64×64 frames were enlarged to 512×512 PNG images while preserving color, appearance, and spatial relationships.
• Lossless visual memory: After each action, the framework stores all returned frames, including intermediate animation frames, indexed by episode and frame number.
• Active visual inspection: Through an inspect tool, the model can retrieve a selected frame, crop and enlarge a region, inspect pixel-level details, or compare several frames before and after an action.

At each step, the model observes the current state and forms a hypothesis about the next action. When the available evidence is insufficient, it can revisit earlier observations or examine a small region in greater detail. After acting, it compares the predicted outcome with what actually happened and updates its understanding of the environment.

VISTA also maintains two text files for long-horizon continuity. GUIDE.md stores rules and experience that may transfer across levels, while WORKING.md records the current state, progress, and next steps. The framework normally exposes only the final frame after an action in the active context; intermediate frames remain in the archive and are retrieved on demand, limiting context consumption.

Results and implications

On ARC-AGI-3, Claude Opus 5.0 with VISTA completed all 25 public games and reached an action-efficiency score of 100. GPT-5.6 Sol also completed all games with a score of 99. With the same GPT-5.6 Sol model, VISTA raised GameWorld success from 40.0% to 63.3%, increased the AI GameStore score from 47.3 to 140.3, and improved BabyVision maze and connection-task accuracy from 41.0% to 63.2%.

The broader implication is that a harness can serve as a visual memory layer, not merely a text-based task manager. Preserving raw evidence and allowing selective inspection may help agents reason about appearance, spatial relationships, and dynamic changes. However, the reported tests focus mainly on games and puzzles. Whether the approach remains reliable in open-ended physical environments is still an open research question.

Source: QbitAI