Safety Benchmarks May Measure Policy, Not Capability

Introduction

A high score on a safety benchmark is usually read as evidence that a guardrail is better at identifying harmful requests or avoiding unnecessary refusals. A study featured in Hugging Face Daily Papers argues that this interpretation can be incomplete. The model may not simply be learning a general property of unsafe text. It may be learning the operational policy encoded in its labels.

One prompt set, two policies

The researchers used an unusual industrial-deployment dataset in which roughly 53,000 identical prompts had been labeled twice. One policy asked whether a prompt was a request for harmful content. The other asked whether a moderator would block it. Those labels were used to train ten guardrail arms sharing the same architecture, codebase, and optimizer family, across two model scales. This setup was intended to hold most engineering variables constant while exposing the effect of labeling policy.

The findings are striking:

• The training pools already contained large portions of two commonly used over-refusal benchmarks. About 88% of JailbreakBench-benign and 82% of OR-Bench-hard appeared in the pools. The subsequent analyses were decontaminated.
• Benchmarks differed sharply in their sensitivity to policy. Across the ten arms, HarmBench-copyright showed a range of 0.99, while WildGuard-benign varied by only 0.043—a roughly 23-fold difference.
• A change involving only about 0.03% of the parameters could move some benchmark scores across their full observed ranges. HarmBench-copyright spanned 0 to 100, while OR-Bench-hard over-refusal ranged from 3.4% to 56.4%.
• The arms receiving a particular policy in their final supervision stage tended to occupy an extreme on the relevant benchmarks, even though the study reports no corresponding change in underlying detection capability.

The hidden influence of teacher models

The issue also appears in distillation. According to the study, a student can inherit a teacher’s policy through soft labels even when all of the student’s hard labels come from the other policy. Teacher provenance therefore becomes an important but often unreported variable in released checkpoints. Two models with similar architecture and training recipes may encode different refusal boundaries because their supervision histories differ.

Validation accuracy provides another warning sign. On a split evaluated under a common grading policy, accuracy correlated more strongly with agreement with the annotator’s policy, at 0.74, than with detection capability, at 0.34. In other words, a model can appear accurate partly because it has learned how the evaluator defines a blockable request.

What this means for evaluation

The study does not imply that safety benchmarks are useless. Guardrails are deployed under concrete product policies, and choosing stricter blocking or lower false-refusal rates can both be legitimate goals. The problem arises when a score produced under one policy is presented as a universal measure of safety capability without revealing the policy behind it.

The authors recommend reporting training-data contamination, a sufficient seed floor, the exact grading definition, and the provenance of teacher models and soft labels. Developers should also show performance across policies and thresholds, including both missed harmful requests and unnecessary refusals. Separating policy alignment from detection capability would make guardrail comparisons more transparent and scientifically meaningful.

Source: Hugging Face Daily Papers