ActionOPD: Distilling Agent Decisions Instead of Tokens

Introduction

Distilling a strong language-model agent into a smaller model is not simply a matter of making the student produce similar text. In interactive tasks, the decisive question is what the agent does at a particular state: which tool it calls, what arguments it supplies, whether it gathers more information, or whether it terminates. ActionOPD, presented in the paper “Distilling Decisions, Not Tokens,” proposes to make this decision the primary unit of distillation rather than copying the teacher token by token.

Why semantic actions matter

Conventional offline imitation learning provides dense, token-specific supervision. Its weakness is that the training states mostly come from teacher-generated trajectories. Once the student starts behaving differently, it may encounter states that are poorly represented in the demonstrations. This state-distribution mismatch can make imitation less reliable.

Token-level on-policy distillation partly addresses the problem by training on states reached by the student itself. However, it commonly depends on access to the teacher’s token probabilities and on compatible tokenization between teacher and student. Those requirements are restrictive when the teacher is accessed only through an API or when the two models come from different families.

ActionOPD moves the alignment target to the semantic action level. At a state encountered by the student, the method samples an action from the student and another from a black-box teacher. It then evaluates how semantically compatible the two actions are and uses that signal in policy optimization. The student therefore learns to make decisions that agree with the teacher without having to reproduce the teacher’s wording or hidden reasoning process.

Core ideas

• Student-induced states: Training takes place on states the student actually visits, reducing reliance on teacher-only trajectories.
• Decision-level supervision: The method focuses on executable choices rather than the exact sequence of generated tokens.
• Black-box compatibility: Teacher token probabilities are unnecessary, and teacher and student need not share a tokenizer or model family.
• Agent-oriented granularity: Semantic actions map naturally to tool calls, environment interactions, and multi-turn task decisions.

Results and implications

The authors train Qwen3-4B and Qwen3-8B with two teachers on diverse, curated tool-use tasks. Across four out-of-distribution benchmarks, ActionOPD delivers average gains ranging from 3.3 to 6.9 percentage points. The paper also reports favorable scaling with both student size and teacher capability.

The broader implication is a change in what agent distillation should preserve. For ordinary text generation, token-level alignment remains useful. For an agent that repeatedly interacts with an environment, however, the teacher’s surface form is often incidental. A smaller model may use different wording or follow a different internal route while making the same effective tool choice. Distilling decisions can therefore reduce interface requirements and make cross-family transfer more practical.

Several questions remain open. Action semantics must be defined carefully, agreement between different actions must be measured reliably, and sampling noise from a black-box teacher must be controlled. Even so, ActionOPD offers a clear design principle: when training an agent, teach it what to choose at important states—not merely what the teacher happened to say.

Source: Hugging Face Daily Papers