Can LLM Agents Learn from Tutorial Videos Without Updating Their Weights?

Introduction

People commonly watch a tutorial, retain its lessons, and apply them later. Large language model agents do not yet have an equally straightforward way to do this. Once training is complete, their parameters are usually fixed, while updating a large model whenever new instructional material appears can be expensive. The paper Can Large Language Model Agents Learn by Watching Tutorial Videos? explores whether an agent can acquire useful knowledge from videos without changing its weights.

The idea behind TAL

The authors introduce Tutorial Agentic Learning, or TAL, as a nonparametric continual-learning framework. Rather than treating a video as a one-off context window, TAL converts instructional content into a persistent, structured tutorialbook memory.

Its process has several parts:

• Knowledge organization: Information from the videos is consolidated into a reusable memory rather than left as an unstructured transcript.
• Adversarial gap probing: The system generates challenging questions grounded in the videos. These questions are intended to expose what the current tutorialbook fails to represent clearly.
• Targeted refinement: The memory is revised in response to the detected gaps, creating an iterative learning loop.
• Reasoning-linked retrieval: At inference time, the agent navigates a tree-based table of contents. Retrieval is therefore connected to the agent’s reasoning path instead of relying only on embedding similarity.

The distinction matters. A summary may preserve the main topic while missing the conditions, exceptions, or tactical details needed to solve a new problem. TAL uses difficult questions as a pressure test for those missing details.

What was evaluated

To study the setting, the researchers created VidLearn, a multimodal question-answering benchmark based on 72 chess tutorial videos spanning 12 openings. TAL outperformed the evaluated baselines and improved monotonically as it consumed more videos. Relative to the corresponding vanilla multimodal language models, the reported gains reached 113% on the training split and 91% on the test split.

The evaluation also included actual gameplay. In head-to-head games, the TAL-based agent reached a stronger opening-phase position than its vanilla counterpart in 69% of the games. The paper further reports an evaluation in a second domain, suggesting that the approach is not intended solely for chess. However, the supplied material does not identify that domain or provide its detailed results, so the cross-domain claim should be treated as an initial signal rather than a complete demonstration of generality.

Why it matters

TAL offers a practical alternative to the assumption that an agent must update its parameters in order to learn. Tutorials can become durable external knowledge, and a structured table of contents gives the agent an explicit route for locating relevant material during reasoning. This could be useful for software instruction, professional training, and other settings where knowledge arrives continuously through demonstrations.

There are also important open questions. A persistent memory can preserve mistakes as well as correct lessons. It may contain contradictions across videos, inherit biases from its sources, or fail when a tutorial leaves key context implicit. Better performance on chess openings does not by itself establish robust, transferable understanding. Future work will need broader domains, longer learning sequences, and more demanding interactive tasks to determine whether tutorial-based memory can support genuinely lifelong agents.

Source: Hugging Face Daily Papers