Validate the Comparison, Not Just the Scorer in LLM Evaluation

Introduction

Interactive LLM studies often test whether conversational pressure changes a model’s behavior. A model may first be asked to reconsider an answer, then receive a more forceful prompt insisting that a fictional entity really exists. Researchers can compare the two sets of responses to estimate how much the pressure matters. An automated scorer then labels whether each answer contains an unsupported factual claim.

The danger is that scorer validation is frequently reduced to one aggregate accuracy or F1 score. A new arXiv study argues that this is not enough. What matters is not only whether the scorer performs well overall, but whether it measures the particular contrast that the study intends to report.

Key findings

• Errors can be prompt-dependent. If a scorer produces more false positives after one prompt than another, the observed difference between prompts may be exaggerated. Unequal false negatives can suppress the effect, and sufficiently asymmetric errors can even reverse its apparent direction.
• Aggregate metrics can hide uneven behavior. In a balanced audit of 288 responses, the scorer achieved an overall F1 of 0.916. Yet, depending on the prompt, it marked between 2.6% and 33.3% of responses as containing unsupported claims when the primary LLM rater judged that they did not. The prompt-level estimates were themselves uncertain.
• Matched cases provide a stronger test. In a second audit, the scorer and two separate LLM raters evaluated responses from the same 300 model-question cases under each prompt condition. A seemingly large 9.3-point increase associated with an assertive certainty prompt, compared with a neutral reconsideration prompt, dropped to 2.3 points under both additional raters.
• A stable direction is not automatically significant. The interval from the primary rater included zero, so the study did not establish that the increase was real. Two other prompt comparisons retained their direction across raters, while an attempted prompt-specific correction remained too uncertain to support a firm conclusion.

Why it matters

The paper shifts evaluation practice from validating a classifier in the abstract to validating the comparison that will appear in the final result. This distinction matters in studies of factuality, refusal behavior, safety, sycophancy, and conversational influence. A scorer may be useful for ranking responses overall while still introducing systematic bias into a particular experimental contrast.

A more defensible workflow would audit both prompt conditions on matched cases, report prompt-level false-positive and false-negative estimates, and include uncertainty intervals rather than relying on one aggregate number. Independent human or LLM raters can provide a useful cross-check, but their own uncertainty must also be carried into the reported effect.

The lesson is not that automated evaluation should be abandoned. It is that the validation target must match the research claim. Before concluding that a prompt changes model behavior, researchers should test whether the measurement process itself changes with the prompt.

Source: arXiv