How to Allocate Agentic RAG Evaluation Budgets

Why budget allocation matters

Evaluating an agentic retrieval-augmented generation system is more complicated than running each question once and reporting an average score. An evaluator can spend resources in at least three ways: cover more questions, execute multiple search trajectories for the same question, or repeat answer generation after a trajectory has been produced. These choices consume different amounts of model computation and can affect both statistical precision and the stability of the final comparison.

This study examines that allocation problem using a retrieval-feedback comparison on HotpotQA and MuSiQue. It evaluates allocation precision, reading efficiency, and cost boundaries rather than treating the total token budget as the only relevant variable.

Main findings

• Broader question coverage is usually the strongest use of a fixed budget. With approximately 34.14–34.39 million model tokens, allocating resources across more questions lowered the standard error by 33% compared with five reads, and by 12.6% compared with three trajectories. For estimates intended to describe overall system behavior, adding cases may therefore be more valuable than repeatedly probing a small set of cases.
• Allocation can be forecast before the full evaluation is complete. Archived nested forecasts and question-only forecasts predicted the eventual allocations within 4.0% and 3.5%, respectively. However, depth-subset analysis found no clear forecasting advantage beyond the two-trajectory audit. More elaborate forecasting is not automatically better.
• One read is not always a major statistical liability. At the same token budget, the variance penalty of using one read rather than the fitted optimum ranged from 0% to 9.9%. The result comes with substantial uncertainty for the Pro condition, so it should not be interpreted as evidence that repeated reads are never useful.
• The cost winner depends on the pricing regime. Under the recorded model fees, more questions outperformed more trajectories when search prices were between $0 and $1 per 1,000 requests. The study does not resolve the fee ranking between adding questions and adding reads, leaving that decision dependent on the actual model and tool pricing.
• Deterministic decoding reduces disagreement, not necessarily comparison quality. Setting temperature to zero reduced answer disagreement from 14.3% to 3.4%, while comparison precision stayed similar. Lower randomness can make an audit easier to reproduce, but consistency alone does not guarantee a more discriminating evaluation.

Implications for evaluation design

The practical default suggested by the results is to expand question coverage first when the goal is a reliable estimate of average system performance. Additional trajectories may be justified when search behavior itself is the object of study, while repeated reads may be useful when answer-level instability is a central concern. The right choice is therefore tied to the evaluation question, not just to the total number of available tokens.

The findings should also be read within their stated scope. They come from two benchmarks, a particular retrieval-feedback comparison, and recorded fee assumptions. The unresolved question-versus-read cost ranking and the uncertainty around the Pro variance estimates argue against turning the results into a universal rule. Future audits will need to combine task difficulty, model pricing, decoding randomness, and the intended deployment objective when choosing how to spend evaluation budgets.

Source: Hugging Face Daily Papers