Teaching Robots Through the World: RoboScript Brings Feedback into Context

Introduction

Robotic manipulation policies often learn by fitting actions to large collections of demonstrations. Once the skill is written into network weights, execution can be efficient, but the policy may also become tied to the scenes, appearances, and object arrangements represented in its training data. A new environment can therefore expose failures even when the underlying task is familiar.

In-context learning offers a different approach: the model keeps its parameters frozen and uses demonstrations or prompts as temporary knowledge. Yet a demonstration recorded in another scene contains no direct feedback about the world currently facing the robot. RoboScript addresses this gap with World In-Context Learning (WICL). Instead of merely showing a model an example, the system lets it rehearse the task in an interactive world and records the consequences in its context.

How the approach works

• Semantic task understanding: The authors argue that in-context learning captures task steps, object roles, and action relationships more readily than scene-specific visual details.
• Rehearsal in simulation: Starting from a single demonstration, a frozen model attempts the task in a simulator or another accessible world. The resulting feedback helps it identify and correct unsuccessful action sequences.
• Task Memory: The accumulated experience is packaged as a Task Memory. This stores task-level knowledge without gradient updates or changes to the foundation model’s weights.
• Deployment data isolation: RoboScript is designed so that information used during deployment is available to the real robot. This separates simulated rehearsal from inaccessible information about a target scene.
• Reflection and embodiment adaptation: The system can improve its memory through reflection and adapt memory across visual distributions and robot embodiments.

Results and implications

On all 50 tasks in RoboTwin 2.0, RoboScript achieves an 83.4% mean success rate across clean and randomized settings. The result uses one demonstration for each task and no gradient updates. According to the supplied paper summary, it exceeds the strongest trained action policy by more than 20 percentage points. The authors also report generalization across visual distributions, improvement through reflection, scaling with the foundation model, and transfer to a physical dual-arm robot through cross-embodiment memory adaptation.

The broader contribution is architectural. Rather than forcing every new experience into the model’s permanent parameters, RoboScript separates foundation-model reasoning, world-based rehearsal, and task memory. The model interprets the task; the simulator supplies opportunities for trial and feedback; the memory preserves what was learned for later deployment. This design could make task-specific adaptation easier to create, freeze, inspect, and reuse.

The reported evidence does not by itself resolve the simulation-to-reality gap, the behavior of very long-horizon tasks, or how task memories scale across a large skill library. Those questions remain important for practical deployment. Still, WICL advances a useful perspective: for embodied agents, learning does not have to mean changing weights. With an interactive world in the loop, context can become a place where a robot acquires and refines manipulation skills.

Source: Hugging Face Daily Papers