AgentJudgeBench Tests Whether LLM Judges Can Reliably Evaluate Agents

Introduction

Evaluating an agent is more complicated than checking whether its final answer sounds plausible. A tool-using system must select the right tool, provide valid arguments, respect dependencies, and recover appropriately when intermediate results affect later steps. Many evaluation pipelines now ask another large language model to judge these traces, but the reliability of that practice has not been well established for structured workflows.

AgentJudgeBench is designed to examine this gap. Instead of treating judging as a general text-preference problem, it focuses on agentic tool-calling tasks represented as dependency-driven directed acyclic graphs, or DAGs.

Key findings

• A multi-dimensional benchmark. The benchmark includes 3,808 instances, six DAG topologies, and three difficulty tiers. The experiments use five task generators and six judge models spanning different capacity levels.
• Difficulty steadily reduces agreement. Alignment with programmatic reference outcomes declines monotonically as task difficulty increases. Without ground truth, the decline is about 1.5 times faster than when reference information is available.
• A scale-independent ceiling appears. On hard queries without ground truth, all six judges cluster within a 77%–82% alignment band. Larger models do not overcome this range, pointing to a structural limitation driven primarily by workflow difficulty.
• Ground truth can backfire. Providing a reference answer lowers alignment for GPT-5.4 by 1.5 percentage points and for Gemini-2.5-Pro by 3.9 points. The authors associate this behavior with possible over-anchoring, where a judge follows the reference too closely instead of independently checking the execution trace.
• Mitigations are uneven. Chain-of-thought reasoning and judge temperature have negligible effects. Structured evaluation rubrics improve alignment by as much as 6.5 points, but the benefit does not transfer consistently across every generator–judge pairing.

Why it matters

The study challenges the assumption that a larger judge is automatically a better judge. In an agent workflow, the evaluation target is not only the final response. It includes tool selection, argument validity, execution order, dependency satisfaction, and the resulting state. A local mistake can propagate through subsequent steps, while a judge looking mainly at the final output may miss where the failure occurred.

A more robust evaluation design should therefore make these dimensions explicit. Tool names, arguments, dependency checks, intermediate outcomes, and final state can be scored separately, with programmatic validators complementing model-based assessment. Ground truth should also be treated as evidence rather than an unquestionable authority, especially when it may encourage anchoring.

AgentJudgeBench does not simply identify one universally superior judge. Its broader contribution is to expose weaknesses in the evaluation layer itself. Developers reporting agent performance may need to describe how the evaluator behaves across workflow topologies, difficulty levels, and ground-truth conditions. Otherwise, an unreliable judge can quietly become the mechanism that defines an agent’s apparent capability.

Source: Hugging Face Daily Papers