HALO Shows Why Proactive Agents Must Learn to Hold Back

Introduction

A proactive agent must often act before a user makes an explicit request. That seemingly simple capability hides several decisions: Is there a real need? Is this the right moment? And how much help is appropriate? Many existing evaluations reduce these questions to task success or a binary judgment, making it difficult to understand why a human would reject an otherwise plausible response.

The paper HALO: Human-Grounded Granular Evaluation of Proactive Agents proposes a human-centered evaluation framework designed to expose these details. Rather than treating proactivity as a single capability, HALO breaks it into observable decisions about evidence, timing, action level, and intervention scope.

Key findings

• Four types of proactive need. HALO-Arena covers Dependency, Blockage, Risk, and Opportunity across 10 everyday domains and 50 personal environments.
• Evidence comes before intervention. Scenarios require agents to retrieve information from structured databases or unstructured documents before deciding whether assistance is warranted and how far it should go.
• Human diagnosis beyond pass/fail. HALO-8K contains 8,000 responses from 10 models across 400 scenarios. Twenty experts annotate action levels, acceptability, and specific reasons for rejection.
• Effort is recorded as well. The benchmark tracks reasoning-token counts and retrieval rounds, offering signals about the work performed before a proactive decision.
• Over-proactivity dominates failures. The best proprietary model reaches 72.88% acceptability, compared with 56.63% for the best open-source model. Premature timing or excessive intensity appears in 61.20% of rejected responses.

Why it matters

HALO reframes the central question for proactive systems. The goal is not simply to make an agent notice more opportunities to help. It is to make the agent intervene at the right time and with the right amount of force. A model may retrieve relevant evidence and identify a genuine issue, yet still fail because it acts before the user is ready or provides a much larger intervention than the context supports.

This distinction matters for products that can remind, plan, monitor, or execute tasks on a user’s behalf. A system that never acts is ineffective, but one that constantly interrupts can also become unusable. The benchmark therefore points toward a more balanced target: evidence-sensitive initiative combined with calibrated restraint.

HALO also complicates the assumption that more reasoning or more retrieval automatically leads to better assistance. Effort can improve the factual basis of a decision, but it does not by itself solve the question of timing or intensity. Future training and safety evaluations may need to optimize these dimensions jointly, rather than rewarding initiative in isolation.

For real-world deployment, the paper offers a practical diagnostic lens. Evaluators should ask not only whether an agent can detect a problem, but also whether it can avoid creating a new one through premature or excessive intervention.

Source: Hugging Face Daily Papers