Back to articles

When Robot Benchmarks Mislabel Success: SimAudit Finds Checker Defects

3 min read

A robot policy’s reported success rate is partly determined by a piece of hand-written code: the benchmark’s success checker. Yet a checker that accepts an expert demonstration has only shown that it recognizes one way to complete a task. A learned policy may stop after partial progress, or it may satisfy the instruction through a different sequence of actions. In those situations, a checker can produce either a “phantom success,” accepting an incomplete execution, or a “silent failure,” rejecting one that fulfills the task. Both errors can distort what a benchmark score says about a policy.

The paper introduces SimAudit, an LLM-based pipeline for examining the relationship between task instructions and checker code. It identifies situations in which the two might disagree, recreates those situations in simulation, and runs the original checker. Multi-view renderings are then used to screen whether the simulated behavior appears to satisfy the instruction. The authors describe visual verification as a filter for potential defects, not as ground truth. They also report that two human annotators independently checked all 61 reported defects. In that validation, they observed no case where visual screening overturned a correct checker verdict; this is not a claim that visual models are infallible.

What the audit found:

  • Across RoboTwin 2.0, RoboCasa365, and RoboDojo, 183 simulated test cases produced the predicted checker verdict.
  • After visual verification and deduplication, the study identified 61 distinct defects across 49 tasks: 40 phantom successes and 21 silent failures.
  • In RoboCasa365, most defects accepted incomplete executions. In RoboDojo, most rejected valid executions. RoboTwin 2.0 showed both types commonly.

These counts describe defects found in the constructed audit cases; they are not estimates of the overall error rate of each benchmark. Their significance is methodological: validating a checker only on expert demonstrations leaves important cases untested. If a checker recognizes only the demonstrated trajectory, a different but valid completion may fail. Conversely, a rule that checks too little may accept an execution that has not actually satisfied the instruction. A more robust evaluation should therefore test both negative examples—tasks that remain incomplete—and valid alternatives to the demonstration.

For robotics research, this shifts some scrutiny from the policy to the instrument used to measure it. A success rate is not automatically an objective account of capability; it depends on whether the task definition and the checker agree. SimAudit offers a structured way to look for mismatches, but its findings do not directly establish how well any particular policy performs, nor do they remove the need for human judgment. Benchmark developers can make reported scores more interpretable by testing and maintaining checkers as carefully as the policies those checkers evaluate.

Source: Hugging Face Daily Papers

Comments

Checking sign-in status...

Loading comments...

Related articles