Let Agents Compare Before Acting: Value Estimation for Long-Horizon Tool Use

Introduction

As language models move from generating answers to completing tasks through external tools, the next action can determine much of the remaining trajectory. A search, database lookup, file operation, or API call may change the task state and alter which actions are useful afterward. In these settings, an agent needs more than a plausible immediate action: it needs an estimate of whether that action will move the task toward a successful final outcome.

The paper “Choosing Before Acting: Comparative Value Estimation for Long-Horizon Tool-Use Agents” studies this problem through Comparative Inference for Tool-use Agents, or CITA. Its central component is a Comparative Inference Model, or CIM, which evaluates alternative tool invocations under the same context before one of them is executed.

Key points

• Moving beyond sparse outcome rewards. Long tool-use traces are often supervised mainly by a final success or failure signal. Because that signal is far removed from individual decisions, assigning credit to a particular call is difficult. CITA seeks a more targeted estimate of the long-term value of the next step.
• Comparison instead of isolated scoring. Logged trajectories usually show only the call that was selected. They do not directly reveal what would have happened if another available tool had been chosen. CITA therefore constructs comparative supervision for alternative calls in the same situation, asking which option is more likely to support eventual success.
• Combining complementary signals. CIM is trained with observed tool behavior, scalable supervision from a Bayesian tool-graph simulator, and semantic judgments produced through LLM-based comparisons. Together, these signals provide information that a single logged trajectory cannot supply.
• Decision support before execution. CIM estimates how likely a candidate invocation is to contribute to final task success. An agent can use that estimate to rank possible next actions rather than relying entirely on the language model’s immediate generation preference.

Why it matters

CITA reframes tool-use training from evaluating a completed trajectory after the fact to comparing possible decisions before they are taken. The approach treats a tool call as part of a changing process, not as an independent classification choice. Its value is especially apparent for tasks involving repeated retrieval, planning, and external operations, where an early mistake can restrict the options available later.

According to the paper’s abstract, CITA consistently improves Tool F1 and task success across three tool-use benchmarks and several backbone language models. Additional analysis reports that CIM learns step-level value estimates that are useful for comparing tool choices. The available material does not provide the numerical gains or isolate the contribution of each supervision source, so those claims should be interpreted in the scope of the reported abstract. Still, the broader direction is clear: a capable agent should not merely know how to act; it should compare where each action is likely to lead before acting.

Source: arXiv