LLM Judges Show Capability Bias, Favoring Stronger Models

Introduction

Large language models are increasingly used as automated evaluators for model training, benchmark testing, and feedback generation. Their speed and scalability make them attractive alternatives to human review. Yet an automated judge is not automatically neutral. Its own capabilities, the capabilities of the system being evaluated, and the scoring rubric can interact in ways that produce systematic distortions.

The arXiv paper Can We Trust LLM Judges studies this problem in absolute scoring settings, which are closer to many practical evaluation workflows than pairwise comparisons. Across four benchmarks and six models, the authors examine 36 judge–examinee combinations and measure both judging accuracy and directional bias.

Key findings

• Judge capability predicts evaluation accuracy. On most models, task accuracy and judging accuracy are strongly correlated, with Pearson correlations of at least 0.90. More capable models are therefore usually better at recognizing answer quality.
• Higher capability does not eliminate bias. Task accuracy is strongly and inversely related to directional bias, with reported correlations of -0.83 or lower. This suggests that some biases may decline as capability improves, but capability alone is not a guarantee of neutrality.
• Stronger examinees receive more lenient judgments. Across judges, more capable examinee models consistently tend to receive more forgiving scores, with correlations of at least 0.83. Simply selecting a stronger judge therefore does not solve fairness concerns.
• Weighted ensembles can improve calibration. The authors introduce calibrated weighted majority voting, which gives judges different influence according to online estimates of false-positive and false-negative rates rather than treating every vote equally.
• Calibration does not require labeled data. A disagreement-based estimator infers judge error rates from patterns of agreement and disagreement among judges. In a simulated setting with shifting task distributions, the label-free method comes within 0.5 percentage points on average of an oracle with perfect error-rate information, while outperforming individual judges and unweighted voting.

Why it matters

The paper reframes the question of whether LLMs can act as judges. The issue is not only whether a judge achieves a high accuracy score, but also whether its errors change systematically as the evaluated model becomes more capable. A single judge may reward overall sophistication rather than the specific property being measured, creating a moving standard for evaluation.

The proposed ensemble offers a practical direction for settings where large amounts of human-labeled calibration data are unavailable. Monitoring disagreement and updating judge weights could make automated evaluation more robust as task distributions change. However, the reported evidence comes from four benchmarks and a simulated distribution-shift experiment. The abstract does not establish that the method will work equally well across all domains, scoring scales, prompts, or production environments. Correlated errors among judges and changes in evaluation criteria remain important concerns.

The broader lesson is not to abandon LLM judges, but to avoid treating one model’s score as an unbiased ground truth. As evaluated systems become more capable, evaluation pipelines may need multiple, monitored, and explicitly calibrated judges.

arXiv