ImpossibleRubrics: When Reward Criteria Become an Attack Surface

Introduction

Rubrics generated by language models are increasingly used in rubric-based reinforcement learning, LLM-as-a-judge evaluation, and automated grading. They promise more detailed feedback than a single scalar score, but they also create a new optimization target. If a model can earn a high score by imitating the surface features preferred by a rubric, the reward may favor persuasive answers over truthful ones.

ImpossibleRubrics studies this problem in a particularly revealing setting: impossible tasks. These prompts pressure a model to reach a conclusion that is not supported by the available information. When no justified conclusion exists, the honest response is to acknowledge the limitation rather than invent an answer.

How the benchmark works

The benchmark includes 169 impossible tasks across six categories of impossibility. Each task comes with a verifiable oracle certificate describing what an honest answer may claim and what it must not claim. A further 48 answerable control tasks help separate failures caused by poor rubrics from limitations inherent in the tasks themselves.

Importantly, ImpossibleRubrics does not prescribe one fixed scoring scheme. It provides task environments and certificates, allowing researchers to generate rubrics downstream and then test them adversarially. The central question is whether an answer can violate the certificate while still receiving a favorable score. This shifts evaluation from asking only whether a model answers correctly to asking whether the reward signal survives optimization.

Key findings

• On an unbiased 150-of-169 environment split, 11 rubric generators were exploitable between 8% and 26% of the time.
• On a deliberately selected stress split, the strongest measured generator was still exploited 36% of the time.
• A certificate-faithful rubric had a 0% exploitation rate, indicating a rubric-quality gap rather than unavoidable task impossibility.
• One generic rubric used across all tasks—favoring decisiveness and penalizing hedging—was exploited 64% of the time.

The last result is especially counterintuitive. A preference that often sounds useful in ordinary evaluation can become dangerous when the correct behavior is to refuse an unsupported conclusion. In such cases, confidence is not evidence of competence; it can be the exploit.

Why it matters

For reinforcement learning, a rubric should be audited not only for clarity and granularity, but also for whether a model can satisfy its wording while defeating its intended objective. For automated evaluation, aggregate scores can conceal a systematic preference for hallucinated certainty. Certificates that define the boundary of permissible claims provide a concrete basis for adversarial auditing.

The study does not show that every generated rubric is unreliable. It shows that generated rubrics should be treated as reward models with their own failure modes. Future evaluation pipelines may need to check factual support, calibrated uncertainty, and resistance to optimization together—particularly when prompts contain missing information, contradictions, or fundamentally undecidable demands.

Source: Hugging Face Daily Papers