A New Protocol Tests Whether Agents Fail from Ignorance or Incompetence

Introduction

When an agent fails a professional task, the failure is often ambiguous. Did the system lack the domain knowledge required to interpret the task, or did it possess the relevant information but fail to plan and execute correctly? This distinction matters in deployment: the first problem may call for retrieval or better context delivery, while the second may require changes to prompting, tools, or the model itself.

The paper “Ignorance or Incompetence? Constructing Knowledge-Gated, Verifiable Tasks for LLM Agents” presents a protocol designed to make that distinction testable.

How the protocol works

• Separate instructions from knowledge: Each task is paired with a compact artifact that may contain private conventions, reference tables, and utility operators.
• Hold the prompt constant: The instruction is byte-identical in the artifact-provided and artifact-withheld conditions, preventing prompt rewrites from becoming a confounding factor.
• Audit for leakage: Construction-time provenance and leak audits are used to check whether the supposedly private knowledge was already exposed through the task or public data.
• Make outcomes verifiable: Deterministic solvers and rule corpora provide exact ground truth for structured tasks. For outputs that cannot be judged by one executable oracle, the authors use named, criterion-level rubrics.
• Repeat the test: A five-trial empirical screen is used to reduce the risk that a one-off run is mistaken for stable knowledge dependence.

What the experiments show

Across 15 calibration tasks, one frontier-agent configuration achieved a 68.0% pass rate when the artifact was available and 0% when it was withheld. The paper also describes a task where a plausible but incorrect artifact produced 0% across five trials. This is an important qualification: supplying context is not enough if the context itself is wrong or mismatched to the task.

After applying the stated five-trial knowledge-gating screen, seven tasks were retained. These findings demonstrate that the protocol can produce tasks with observable, testable dependence on a knowledge artifact. They do not demonstrate that the retained tasks improve post-training, and the authors explicitly limit their conclusions accordingly.

Why it matters

Benchmark scores become easier to interpret when knowledge access is treated as an experimental variable rather than an uncontrolled background condition. A failure that disappears once the relevant artifact is supplied points toward retrieval, context packaging, or knowledge management. A failure that persists despite correct knowledge may instead reveal weaknesses in planning, tool use, instruction following, or execution reliability.

The protocol also highlights several limitations. The calibration results are configuration-relative, the evaluated set is limited, and the strength of the gate depends on artifact design and evaluation criteria. The released suite is only partly public, so broader replication remains necessary. Real deployment logs and longer-horizon workflows would be especially useful tests of whether the distinction transfers beyond controlled tasks.

Overall, this is not a new agent architecture. It is an evaluation framework for diagnosing why agents fail, making benchmark conclusions more actionable than a single undifferentiated success rate.

Source: Hugging Face Daily Papers