ClawProBench: Evaluating Whether AI Agents Actually Execute Correctly

Introduction

Once an AI agent can read and write files, call tools, send messages, schedule tasks, or launch subagents, its final answer no longer tells the whole story. An agent may reach the right result through unsupported evidence, an incorrect tool route, or an unsafe action. It may also succeed once and fail on the next attempt. ClawProBench addresses this problem by evaluating a declared model-plus-runtime configuration and preserving the full execution trace rather than reducing every trial to a final-answer label.

What the benchmark changes

• The unit of evaluation is broader. The benchmark distinguishes failures in evidence acquisition, runtime routing, safety boundaries, and repeated execution. Two systems with the same final answer can therefore receive different assessments if one takes a riskier or less reliable path.
• It is runtime-native. Built around OpenClaw, the benchmark covers workspace tools and native surfaces for browsing, memory, messaging, scheduling, skills, and subagents.
• It has two complementary tracks. The full profile contains 102 scenarios involving live workspaces and native-runtime routing. The frozen holdout contains 68 scenarios with closed-world JSON output contracts, making it more suitable for robust ranking.
• Scoring is process-aware. Execution traces are used to combine correctness, process quality, and efficiency under a safety gate. Failure evidence is retained for inspection instead of being discarded after scoring.
• Repeatability matters. On the holdout, pass@k-any reaches 0.6638, while strict success across three trials is only 0.2890. A system that succeeds at least once is therefore not necessarily a dependable system.

Results and interpretation

The study evaluates 68 configurations on the full profile and 37 on the holdout. The highest safety-gated average trace score on the full profile is 0.7671. Workspace-live scenarios average 0.6415, compared with 0.5238 for native-runtime tasks. The difference suggests that routing through the runtime’s native capabilities creates additional failure opportunities, including state management and tool selection.

The two tracks also produce weakly aligned rankings: their Spearman correlation is 0.1300. Rankings based only on correctness differ substantially from rankings that account for process quality, safety gating, or strict repeated success. This is a useful warning against treating a single aggregate accuracy number as a complete measure of agent capability.

Why it matters

ClawProBench’s main contribution is methodological. It argues that agent benchmarks should specify not only the model, but also the runtime, tools, permissions, state, and repetition protocol under which the model operates. Trace data can help developers determine whether a failure came from reasoning, evidence collection, tool routing, or authorization. For deployment teams, repeated success is more informative than a lucky pass. For benchmark designers, a frozen holdout can provide a more stable basis for comparison than a set of tasks that systems may have indirectly optimized for.

The results should still be interpreted within scope: the benchmark is instantiated on OpenClaw, so its measurements are tied to that runtime’s interfaces and assumptions. Broader comparisons will require reproducible environments and shared safety criteria across runtimes. Even with that limitation, the central lesson is clear: an agent should be judged not only by whether it produces the right answer, but also by whether it reaches that answer safely, efficiently, and consistently.

Source: Hugging Face Daily Papers