Can Multimodal Agents Act on What They See? EASEL Puts Precision to the Test

Introduction: Seeing Is Not the Same as Doing

Multimodal models can answer questions about images and increasingly interact with browsers, graphical interfaces, and software tools. Yet those abilities do not necessarily mean that a model can carry out a delicate task from visual feedback. In a precision setting, the agent must estimate positions, colors, sizes, and trajectories, execute an action, inspect the new state, and then adjust its next move. A small error can propagate through the rest of the task.

The paper Paint What You See describes this capability as dexterous visual tool use and introduces EASEL as a benchmark for studying it. Rather than asking for a single tool call, EASEL places the agent in an incremental painting environment. The goal is to reproduce a reference image through a sequence of parameterized actions.

Key points

• Reference-guided reconstruction is the main test. The agent must select painting actions from visual evidence and repeatedly revise the canvas. Tool parameters directly affect the quality of the final output.
• The benchmark goes beyond painting. EASEL also includes semantic tasks for region annotation, handwriting, and path planning. These tasks probe precise localization, trajectory control, and structured visual action.
• The main weakness is not simply image recognition. Evaluation of 25 models reports reconstruction similarity in the relatively low range of 0.40–0.54. Trajectory diagnostics show that agents often improve early, reach a local peak, and then lose quality as later actions accumulate errors.
• Specialized trajectory data helps, but does not solve the problem. EASEL-Data contains 440,000 samples organized as a two-stage curriculum for trajectory supervision. EASEL-9B improves over its base model by 6.3% relatively and ranks third among the evaluated models.

Why it matters

Many agent benchmarks focus on whether a high-level task was completed: a page was opened, a form was filled, or code was produced. Those metrics are useful for planning, but they can hide whether the agent can translate visual judgments into stable, corrective actions. EASEL evaluates both the final artifact and the path taken to create it, separating “understanding the objective” from “executing it reliably.”

This distinction helps explain why strong image-question answering does not automatically transfer to precise annotation or path planning. In these settings, visual feedback must be part of an active control loop, not merely context supplied at the beginning. The agent needs to detect its own mistakes and recover before they compromise the final result.

The results also offer a measured view of training. Curriculum-based trajectory supervision produces a clear gain, suggesting that this capability can be learned rather than treated as an entirely emergent property. At the same time, a 6.3% relative improvement leaves open questions about error recovery, long-horizon stability, and generalization to unfamiliar tools.

For GUI agents, creative software assistants, and embodied systems, EASEL points toward a broader evaluation principle: an agent should not only know what the target looks like or which tool to call. It should be able to watch the consequences of its actions and keep acting accurately. As multimodal systems move from answering questions to changing environments, benchmarks built around that closed loop will become increasingly important.

Source: Hugging Face Daily Papers