LLM-as-a-Coach: Turning Feedback into Transferable Experience

Introduction

For open-ended problems, standard reinforcement learning often squeezes a model’s feedback into a single scalar reward. That is convenient for optimization, but it also discards a lot of useful textual nuance. The paper behind LLM-as-a-Coach argues that non-verifiable tasks deserve a richer learning signal: not just a score, but reusable experience.

Core idea

The method called Experiential Learning (EL) repurposes the usual LLM-as-a-Judge setup into an LLM-as-a-Coach. The coach does more than assign a rating. It distills its assessment of each on-policy response into transferable experiential knowledge, which then conditions a teacher model and is internalized by the policy through on-policy context distillation.

In plain terms, the system keeps the reasoning behind feedback instead of collapsing it into a number. That gives the policy denser supervision and preserves distinctions between high-quality outputs that a scalar reward would treat as nearly identical.

Why this matters

Non-verifiable tasks are exactly where scalar rewards become fragile. Creative writing, open-ended dialogue, planning, and other subjective tasks often have multiple acceptable answers, each with different strengths. A single score can hide those differences and even encourage shortcut behavior.

Key takeaways

• Higher-bandwidth supervision: text-based experiential knowledge carries more detail than a reward scalar.
• Better separation among good answers: EL can preserve fine-grained preferences that standard RL may blur.
• Improved generalization: the method reportedly works better on held-out and unseen open-ended tasks.
• Less reward hacking: richer feedback makes it harder for the policy to exploit the objective.
• Works across setups: the paper evaluates two policy families and uses feedback from both the policy itself and a proprietary model.

Implications

The broader message is that post-training may need a new unit of learning. Instead of asking only, “How good was this answer?”, we may want the system to capture “What exactly was learned from this answer?” That shift turns evaluation into a more reusable training resource.

If this direction holds up, it could make alignment and post-training on subjective tasks more robust. The coach-style interface may also help models improve without overfitting to brittle reward signals.

Hugging Face Daily Papers