RoboValue Tests Whether Robotic Value Models Truly Understand Execution

A robot that completes a task successfully is not necessarily a robot that can explain whether it is progressing correctly along the way. For policy learning and execution monitoring, robotic value models are expected to turn visual observations, task instructions, and execution history into useful feedback. They should recognize the current state, estimate progress, detect failures, and judge whether a recovery attempt is working.

Many existing evaluations, however, focus on selected aspects of execution understanding. Final success or a general progress score may hide errors in intermediate judgments. RoboValue addresses this gap with a unified, fine-grained benchmark spanning simulated and real-world manipulation.

Four dimensions of evaluation

RoboValue organizes assessment around four complementary capabilities:

• Task-state understanding: whether a model can identify what state the task is in from visual observations and instructions;
• Temporal progress monitoring: whether it can use execution history to distinguish progress, stagnation, and deviation;
• Failure and recovery reasoning: whether it can recognize a failure and determine whether subsequent actions are actually recovering the task;
• Value consistency: whether value judgments remain coherent when instructions or valid execution orders change.

The benchmark does not merely ask whether a trajectory ends successfully. Diagnostic trajectories and instruction variations are designed to test whether the assigned values reflect task requirements at different moments. This makes the evaluation closer to the role value models play in reinforcement learning: shaping decisions before the final outcome is known.

A shared sim-to-real setup

RoboValue-Dataset contains 15 simulation tasks and 20 real-world manipulation tasks. It includes expert demonstrations for training and annotated trajectories for fine-grained evaluation. The project also releases Full-Shot and One-Shot training subsets in LeRobot v3.0 format, making the data easier to integrate into existing robotics pipelines.

The authors evaluate representative value models in Zero-Shot and One-Shot settings. They also separate shifts caused by changes in robot embodiment from shifts caused by changes in the environment. This distinction matters: a model may understand the task conceptually yet struggle when the robot body, viewpoint, or surrounding scene changes.

What the results reveal

The reported findings point to weaknesses that broad outcome metrics can miss. Models with strong outcome and forward-progress scores can still make mistakes when grounding fine-grained instructions, tracking progress that depends on prior events, or assessing recovery after failure. The same semantic subtask may also receive substantially different value gains under different valid execution orders.

This suggests that value models do not always connect an action with its task meaning in a stable way. A step that looks productive in isolation may not advance the overall objective, while an apparently repetitive action may be valuable in the context of recovery. If such distinctions are missed, policy optimization can reinforce superficial behavior rather than robust task completion.

RoboValue is therefore useful not only as a leaderboard but also as a diagnostic framework. It identifies where value supervision needs improvement and offers shared interfaces for comparing models across embodiments and environments. The benchmark cannot replace testing in deployment, and broader task diversity and longer interactions will remain important future challenges. Still, by asking whether a model can judge the entire execution process consistently—not merely the final result—RoboValue sets a more demanding standard for robotic feedback.

Source: Hugging Face Daily Papers