GPT-Policy: In-Context Learning for Adaptive Robots

Introduction

A robot operating in an unfamiliar environment must do more than recognize objects or produce a plausible motion. It needs to use demonstrations and interaction feedback to adjust its behavior after deployment. Conventional robot policies generally depend on large collections of training data and task-specific optimization, yet no finite dataset can cover every combination of objects, states, and environments a robot may encounter. The paper In-Context Robot Learning with VLM Agents introduces GPT-Policy as an attempt to use the in-context capabilities of vision-language models for this problem.

How GPT-Policy works

The framework contains three main components:

• A context compiler preserves visual transitions that are relevant to the task, turning demonstrations and interaction history into a more focused context.
• A vision-language model agent interprets the current observation and available context, then proposes actions involving the robot and its tools.
• A constrained controller checks whether an action can be executed, carries it out, and reports the outcome back to the agent.

The division of responsibilities is important. The VLM is not asked to directly control every low-level degree of freedom. Instead, it uses demonstrations and feedback to reason about the next useful operation, while the controller enforces execution constraints and closes the loop with the physical system. The process does not require gradient updates or persistent task-specific parameter changes, making it closer to deployment-time adaptation than conventional policy training.

What the experiments show

The authors examine task success, efficiency, comparisons across models, and controlled ablations of the available context. In real-robot trials, human video demonstrations improve task completion even when they do not provide explicit robot action labels. This indicates that visual changes in object state, action order, and outcomes can contain useful task information without a direct mapping to robot commands.

However, the gap between watching a human and controlling a robot becomes more visible in contact-sensitive situations. Human motions and robot embodiments are not identical, and a visually reasonable action may still be difficult to execute safely or precisely. The study reports further gains when the context includes action references aligned with the robot’s behavior, particularly for tasks involving contact.

Significance and limitations

GPT-Policy offers an alternative to fully end-to-end robot policy training. A VLM can act as a general reasoning and context-using agent, while a separate controller translates its suggestions into constrained physical behavior. This separation may reduce the need for collecting task-specific labels and retraining a policy whenever a new task appears. It also provides a practical interface between video demonstrations, feedback, and robot execution.

The framework should not be interpreted as a complete solution to general robot learning. The model still has to understand demonstrations correctly, ground abstract intentions in the robot’s capabilities, and recover when execution diverges from expectation. Contact dynamics, embodiment differences, context compression, accumulated errors, and the trade-off between success and efficiency all remain important deployment concerns. GPT-Policy is therefore best viewed as an empirical step toward using VLMs as adaptable robot policy agents, while also making clear how much remains to be solved before such systems can be broadly reliable.

Source: Hugging Face Daily Papers