SO-Arena Looks Beyond Accuracy to Test System-One Reliability
Introduction
System-One models are designed to make fast, general-purpose decisions without producing long chains of tokens. That promise is attractive for interactive products and agentic systems, where latency matters. Yet speed alone does not answer a more practical question: if a model is correct once, will it remain correct when the surrounding options change slightly? And can it recognize that none of the available choices is valid?
SO-Arena is proposed as a benchmark for examining those questions. Instead of reducing model quality to a single aggregate score, it separates the breadth of what a model can do from how dependable its decisions are.
What the benchmark measures
- A broad capability spectrum. SO-Arena spans 11 domains organized into five capability families. This design aims to expose uneven strengths and weaknesses that an average accuracy number can conceal.
- Evidence-grounded evaluation. The benchmark uses reliable sources and manually checks reference labels against the inputs visible to the model. This helps limit errors caused by unsupported labels or information unavailable during evaluation.
- Fine-grained difficulty. Domain-specific criteria are used to distinguish levels of capability, making the evaluation more informative than a simple right-or-wrong tally.
- Interactive sandbox tasks. In selected settings, models must execute actions iteratively rather than merely select an answer once. The setup makes it possible to inspect behavior across action, feedback, and subsequent decisions.
- Two views of reliability. One measure tests decision stability when the option set is changed in a controlled way. The other checks whether a model can recognize that no valid option is available instead of forcing a choice.
- A bridge to System-Two comparison. Reasoning-intensive tasks also support comparisons between fast System-One behavior and approaches that use more extensive reasoning.
Why the findings matter
Results from 21 models suggest that scaling and additional reasoning do not produce uniform gains across domains. A model may benefit substantially in one type of task while showing a smaller improvement elsewhere. This makes it risky to treat larger models or longer reasoning as universal solutions for decision quality.
The most important finding is the separation between performance and reliability. The best-performing System-One model overall ranks only seventh on the reliability measure. In practice, a model can be frequently correct yet still change its decision under a minor option-set adjustment, or confidently select an answer when every option is wrong. Such failures are especially difficult to detect in automated workflows because a single successful output may create a misleading impression of trustworthiness.
SO-Arena therefore offers a useful shift in evaluation priorities. Developers assessing fast decision models should report not only accuracy, but also the distribution of capabilities across domains, performance in interactive environments, stability under controlled perturbations, and the ability to abstain or identify missing valid answers. These dimensions provide a closer view of where a model can be safely relied upon—and where its speed may conceal fragility.
Source: Hugging Face Daily Papers
Comments
Checking sign-in status...
Loading comments...