CLASHarness Turns Multi-Agent Conflict Evaluation into a Verifiable Task

Introduction

When several vision-language models operate files, websites, or business applications together, the central question is not simply whether an agent can follow an instruction. It is whether the agent knows which source to trust. One sub-agent may report a stale state, another may issue a contradictory command, and multiple roles may compete for a shared file, lock, or permission. A reliable system must treat peer outputs as claims to verify rather than as ground truth.

CLASHarness is designed to evaluate that ability. The paper argues that many existing computer-use benchmarks describe clean, cooperative environments with consistent information, leaving a major gap between benchmark behavior and real multi-agent workflows.

Key ideas

• From authored stories to verifiable tasks: CLASHarness combines hierarchical scenario planning with an eight-step verified micro-pipeline. It also provides 23 modular conflict and challenge plugins for situations such as stale information, contradictory instructions, and shared-resource contention.
• Commit facts before rendering: Decisive visual facts are fixed before the interface evidence is rendered. This design aims to limit narrative drift, answer leakage, and rubrics that cannot actually be satisfied.
• Derive graders from execution: The system runs reference trajectories in a sandbox and compiles hybrid graders from them, combining deterministic code checks with semantic judgments. The authors describe the resulting tasks as solvable by construction.
• A multimodal benchmark at scale: The release contains 600 executable multi-round tasks across 20 operational domains, organized into 60 packages, with 7,970 rendered 4K visual evidence assets and 3,284 required checks.

What the evaluation shows

The paper evaluates 15 frontier models and highlights a fundamental capability gap. When a sub-agent's report conflicts with the current environment, weaker models are more likely to follow the misleading report directly. Stronger models are more likely to inspect visual evidence and verify the state before deciding what to do. This suggests that multi-agent collaboration depends on more than language coordination: agents must also reconcile sources, track information freshness, and ground actions in observable evidence.

Why it matters

CLASHarness is important because it turns conflict resolution from a loosely written scenario into a reproducible interactive evaluation. Pre-committed facts, sandboxed reference execution, and automatically compiled graders can reduce the cost of manual task authoring and make model comparisons easier to interpret.

For deployed systems, the lesson is practical. Improving message passing alone is not enough; agents should be encouraged to inspect the shared environment, question stale information, and pause when permissions or states disagree. The reliability of multi-agent computer use may ultimately depend on maintaining a productive balance between cooperation and skepticism.

Source: Hugging Face Daily Papers