LLM Agents Need More Than Accuracy: They Must Know When They May Be Wrong

Introduction

When retrieved evidence conflicts with a language model’s prior knowledge, the important question is not only whether the agent produces the right answer. It is also whether the agent recognizes that its confidence should change. Will it revise its conclusion, explain that the evidence is inconsistent, or continue with its original belief? The paper Accurate but Not Humble: Evaluating Epistemic Humility in LLM Agents under Knowledge Conflict examines this often-overlooked behavior.

Most agent evaluations emphasize whether a task was completed successfully or whether the final answer was correct. Such metrics say little about what happens when an agent encounters contradictory information. The authors therefore study epistemic humility (EH), defined as an agent’s willingness and ability to recognize, respond to, and communicate uncertainty during execution.

Key findings

• ISE turns humility into observable behavior. The framework evaluates three dimensions across an execution trajectory: Identify, detecting a conflict; Solve, taking steps to verify, reconcile, or resolve it; and Escalate, clearly reporting unresolved uncertainty to a user or a higher-level process.
• Two conflict settings are tested. In controlled conflict, the model’s parametric knowledge disagrees with retrieved evidence. In naturally occurring conflict, multiple contextual sources disagree during a multi-step agent run. Both settings are paired with matched no-conflict controls.
• Accuracy and humility can diverge. Across four agents, some configurations recognized conflicts during execution but failed to communicate unresolved uncertainty in the final response. In some cases, the result was an incorrect answer delivered with undue confidence.
• Early detection does not guarantee later resolution. Trajectory analysis shows that agents frequently identify a contradiction in an early step, then lose track of it during subsequent reasoning, tool use, or answer construction.
• Improvement involves a trade-off. Model-level interventions can raise EH, but often at the expense of task accuracy. This suggests that humility emerges from the interaction of the backbone model, the agent harness, and the evaluation environment.

Why it matters

The study turns “knowing when not to be certain” into an evaluable property of agent systems. In high-stakes settings such as research, legal work, healthcare, or enterprise operations, an occasional mistake is not the only concern. A more serious failure occurs when an agent encounters conflicting evidence but suppresses that conflict in its final answer.

The findings argue for evaluations that go beyond outcome scores. Future benchmarks could separately track conflict detection, evidence checking, state preservation, and escalation. They should also distinguish between an agent that is correct but cannot explain relevant uncertainty and one that is wrong yet transparently reports unresolved evidence. At the same time, greater caution should not be reduced to producing more disclaimers. A useful agent must take meaningful steps to investigate uncertainty while still completing ordinary tasks. Reliability may therefore depend less on constant confidence or constant hesitation than on the ability to update beliefs as evidence changes and clearly preserve what remains unresolved.

Source: Hugging Face Daily Papers