A Poor Actor Can Still Be a Strong Teacher for Long-Horizon Agents
Introduction
Long-horizon agents need to do more than select the next action. They must also determine whether completed actions were useful and, ideally, provide feedback that helps another policy learn. These capabilities are often treated as different uses of the same decision model. New work argues that they should be evaluated separately: a model can act poorly while still judging actions well enough to teach.
The study examines three roles. An Actor chooses an action from the current state. A Judge evaluates actions after observing the whole trajectory. A Teacher scores candidate actions to generate step-level signals for reinforcement learning. The experiments use ALFWorld for the actor and judge comparisons, and ALFWorld together with WebShop for the teaching evaluation.
Key findings
- Actors are vulnerable to short-term attraction. Three decision models and three general LLMs were tested on ALFWorld. As actors, all six were drawn toward an easier subgoal and skipped another subgoal that had to be completed first. The result exposes a gap between immediate plausibility and long-term task structure.
- Full-trajectory hindsight changes evaluation. When the same models acted as judges and could inspect the complete episode, they credited actions that advanced the prerequisite subgoal. The consequence of an action, visible after the episode, supplied information that was unavailable during online execution.
- Progress questions are more informative than outcome forecasts. Asking whether an individual action helped the task separated useful and useless steps more effectively than asking whether the entire task would eventually succeed. This gives supervision a more local and actionable target.
- Teaching performance can exceed acting performance. The proposed JEV supplies step-level feedback for training a Qwen2.5-1.5B policy. Against GRPO, success on ALFWorld rose from 72.8% to 82.9%, while WebShop success increased from 56.8% to 71.4%.
Why it matters
The central lesson is that action selection and action assessment face different information conditions. An actor must anticipate consequences from a partial state, whereas a judge can use what actually happened. It is therefore misleading to assume that the best executor must also be the best teacher.
For long-horizon agent training, this opens a practical design space. Teacher models may be selected for the quality and consistency of their intermediate feedback rather than for their standalone task success. Step-level supervision can also make reinforcement learning less dependent on a sparse final reward.
The results do not eliminate the need for caution. Hindsight-based judgments may still be noisy, and the reported evaluations do not establish how well the approach transfers to every environment or longer task. Still, the actor–judge–teacher distinction provides a useful framework for analyzing agent capabilities: execution, evaluation, and supervision can be related, but they are not interchangeable.
Source: Hugging Face Daily Papers
Comments
Checking sign-in status...
Loading comments...