In-Context Learning for Robots: From Demonstrations to Transferable Action

Introduction

A robot can complete a task without actually preserving what a demonstration was meant to teach. A placement trajectory, for example, may encode more than an end position: it may require a particular order, posture, clearance condition, or interaction style. The central challenge is therefore not simply whether a robot can act, but whether it can identify the requirements supplied by a demonstration and realize them when objects, surroundings, or execution conditions change.

The survey In-Context Learning for Robots: Methods and Applications examines this challenge through the lens of robot in-context learning (ICL). In this setting, the robot keeps its neural parameters fixed during deployment and uses demonstrations or interaction to direct existing competence toward a new task. This makes robot ICL distinct from conventional retraining, and also from related approaches that rapidly modify model parameters.

Four interfaces between context and execution

Rather than grouping methods only by model architecture, the survey organizes them according to the interface through which contextual evidence affects physical behavior:

• Context-conditioned policies take demonstrations, observations, or interaction history as conditions for an action policy. The important question is whether the policy preserves the specific requirement conveyed by the teaching, rather than merely producing a plausible action.
• Geometric demonstration transfer maps trajectories, motion references, or spatial relations from a demonstration to new objects and scenes. Its success depends on establishing reliable correspondence between the demonstrated situation and the new one.
• World-model-based control uses contextual information to predict possible futures and then selects actions based on those predictions. The world model serves as a bridge between interpreting the demonstration and controlling the robot, making its treatment of environmental change and action consequences especially important.
• Skill- and agent-based execution converts context into callable skills, programs, or higher-level plans. This interface is naturally suited to compositional tasks and can incorporate memory, recovery, and reuse of prior experience.

These families can overlap in practice. They differ mainly in what carries the teaching signal into execution: an action distribution, a motion reference, a predicted future, or a skill and program structure. The classification exposes the gap between understanding a requirement and physically realizing it.

Why evaluation needs more than success rate

The survey argues that task completion alone cannot establish whether a robot learned from teaching. Evaluation should separate at least three questions: Did the robot respond to the requirement supplied by the demonstration? Can its executor preserve that requirement when the physical situation changes? Does retained memory or experience improve later learning and execution?

This perspective places teaching responsiveness, faithful transfer, physical generalization, memory, correspondence, recovery, and experience reuse at the center of experimental design. These are not peripheral system features. They determine whether context remains useful after the original demonstration has disappeared.

Significance and broader implications

The survey’s main contribution is to frame robot ICL as an end-to-end systems problem. Context must be represented, matched to a new situation, passed into planning or control, and reused when execution fails or a related task appears later. The discussion also connects compositional task acquisition with physical recursive self-improvement: experience should not only help a robot solve the current task, but also improve its ability to learn subsequent tasks.

For researchers, the taxonomy offers a basis for designing more discriminating experiments. For system builders, it is a reminder that average policy performance is not enough. A useful robot must turn demonstrations into action that is faithful, physically transferable, and increasingly effective through experience.

Source: Hugging Face Daily Papers