RoboJev Turns a Vision-Language Model into a Robot Task Judge

Robot learning systems need reliable labels for successful and failed attempts, yet many manipulation datasets contain videos without an explicit outcome annotation. RoboJev addresses this missing layer by acting as a visual-language judge: it does not control the robot, but examines the task goal and the observed result to decide whether an episode succeeded.

How it works

RoboJev-27B is built from JEV-27B-VL. Its inputs are a task instruction, 10 sampled frames from a manipulation attempt, and, when available, final frames from auxiliary cameras. The model predicts success or failure through a calibrated two-token readout, allowing both probabilities to be obtained in a single forward pass.

The adaptation uses a rank-32 LoRA module and updates only the language-model side while keeping the vision encoder frozen. The training procedure also adds a source-specific KL penalty for research data. This regularizes predictions toward the original model on those sources rather than allowing the adapted model to move without constraint. In total, training uses 52,598 episode records.

Reported results

On 268 held-out episodes involving the same SO-101 robot model, RoboJev reaches 95.2% balanced accuracy and a 0.993 AUROC. Its performance remains meaningful when the data comes from previously unseen labs or robots: the model records an AUROC of 0.890 across 459 episodes. On clearly labeled community-collected episodes, it achieves 93.3% accuracy over 135 examples.

The comparison with the base model is also informative. Using the same inputs, the base model reaches a 0.900 AUROC on the unseen-lab/robot subset. This suggests that some judging ability is already present in the underlying vision-language model, while fine-tuning improves or reshapes its behavior for the specific evaluation task. It also shows why cross-source testing matters when reporting robot benchmarks.

Inference is relatively compact for a 27B model: on one H100, serial processing takes a median of 0.26 seconds per episode. The authors estimate roughly $0.19 per 1,000 predictions at a rate of $2.69 per GPU-hour, excluding media preprocessing and serving costs. The release includes inference code and a transparent evaluation package containing 20,628 predictions organized into 29 model or protocol entries.

Why it matters

RoboJev makes an important dataset operation explicit: converting unstructured robot videos into outcome labels. Such a judge could help filter demonstrations, audit historical runs, or support offline monitoring without requiring a human to review every episode. Open weights and released evaluation materials also make the approach easier to inspect and reproduce.

The results should not be read as a universal guarantee of robot understanding. The judge relies on a small set of visual frames and optional final views, so occlusion, camera changes, long-horizon dependencies, and tasks requiring tactile evidence may remain difficult. The performance gap between familiar and unseen sources further suggests that deployment should include calibration for the task, camera setup, and data distribution rather than relying on one headline metric.

Source: Hugging Face Daily Papers