JevAgentHarness Lets Small Models Share Control with LLM Agents

Introduction

Most LLM agents invoke a large model at every step: they inspect the current state, select an action or tool, observe the result, and repeat. This generality comes with a price. On long tasks, repeated calls add latency and make inference expensive, even when the next step is essentially a multiple-choice decision. JevAgentHarness explores a hybrid alternative: let a smaller decision model handle routine choices, while the LLM remains available whenever the smaller model is uncertain.

What the study contributes

• Evaluation beyond isolated questions. The authors build AgentDecisionBench to test decision models at several levels, from standalone questions to public benchmarks and a running agent in the DecisionGym environment. This is intended to measure whether a model can improve the whole control loop, not merely answer local prompts.
• Local accuracy is not enough. The best model on isolated decisions completed only 69.5% of agent tasks. A wrong tool choice can alter the state that follows, causing downstream errors that are invisible in a one-step score.
• Confidence-aware delegation. Without a fitted confidence threshold, delegating tool choices reduced success by 19.0 percentage points. JevAgentHarness fits thresholds for each choice using the agent’s own trajectories. Confident decisions go to the small model; uncertain ones are sent back to the LLM.
• Training on post-error states. Existing decision models may not encounter the unusual states created by their own mistakes. AgentDecision-4B is trained on such states, using Jev’s decisions as labels, so that its input distribution better resembles actual agent execution.

Results and implications

On tasks unlike the training data, agents completed 91.5% of tasks with AgentDecision-4B and 82.0% with Jev. On tasks similar to the training distribution, 4B models trained in this way reduced agent cost by 81.1% to 89.0% and made execution up to 2.5 times faster, without losing 3 or more percentage points of task success. The supplied material does not include the full experimental configuration, so these figures should not be treated as universal guarantees across agents and environments.

The broader lesson is that model substitution alone is not the solution. A practical system needs a policy for deciding when delegation is safe, when the LLM should take over, and how to learn from the states produced by earlier errors. For agent builders, evaluation should therefore combine end-to-end success, calibration, cost, and latency instead of relying on static decision accuracy. The approach could make hybrid agent architectures more economical, but its reliability across new tools, environments, and failure modes remains an important open question.

Source: Hugging Face Daily Papers