OSReward Sets a Standard for Evaluating Computer-Use Reward Models

Computer-use agents are becoming central to how models interact with digital environments, but their progress depends on a question that has not been examined carefully enough: can the models we use as judges actually judge well? OSReward tackles that problem by turning trajectory verification into a standardized evaluation task.

Key takeaways

• A human-gold benchmark across platforms: the dataset covers web, Windows, Ubuntu, and mobile trajectories, collected from diverse agent backbones and labeled through multi-stage human annotation.
• A harder test set for real failures: OSReward-Hard concentrates on genuinely difficult cases where shallow heuristics are likely to break.
• Fine-grained scoring support: OSReward-Multi enables more detailed assessment of efficiency and alignment, not just binary success or failure.
• A large-scale judge audit: the paper’s evaluation finds that even strong VLM judges fall short of ideal performance and share a leniency bias that often turns failed runs into false positives.
• An open training corpus and models: to close the gap, the authors release OS-Shepherd-100K and train open-weight OS-Shepherd 9B/35B reward models on it.

Why it matters

This work is important because CUA evaluation is not just an academic detail; it is the reward signal behind benchmarking, data curation, and reinforcement learning. If the judge is weak, the whole feedback loop becomes noisy and training decisions drift away from reality.

The paper also highlights a practical tradeoff the field has been living with: the models that are reliable enough to trust are often too expensive to deploy at scale, while affordable open models still lag behind. OSReward does not magically remove that tradeoff, but it gives the community a clearer way to measure it and a concrete route toward cheaper, specialized judges built from open data.

For teams building computer-use agents, this is a useful reminder that scaling agents requires scaling evaluation first. In other words, before asking how to make an agent act better, we need to know how to judge its actions better.

Source: Hugging Face Daily Papers