Agent as Policy Lets General-Purpose AI Control Real Robots

Introduction

A robot must do more than recognize a task: it must turn an interpretation into a sequence of physical actions and recover when reality differs from its expectations. Conventional manipulation systems often rely on task-specific data, carefully tuned environments, or separately trained policies. A study titled Agent as Policy (AGP) explores a different design. It connects a general-purpose coding agent directly to a physical robot and gives the agent responsibility for both planning and execution.

The agent receives visual evidence through the robot interface, writes executable programs, sends motion commands, and observes what actually happened. It can then revise its code or subsequent actions instead of following a fixed trajectory from start to finish.

Key points

• Planning and execution are combined: AGP is not only a language-based planner that hands a finished plan to another controller. The agent is expected to translate its reasoning into programs and robot commands.
• The loop remains active at runtime: Physical outcomes become part of the decision process. Position errors, unsuccessful contacts, or unexpected object states can prompt a new action.
• No per-task policy training: The paper emphasizes that model weights remain fixed and that the system does not receive task-specific or environment-specific training for each evaluation.
• A broad manipulation set: The experiments include assembly from human videos, block construction from target images, die flipping or reorientation, targeted throwing, and bimanual towel folding. Together, these tasks span precision manipulation, dynamic motion, and deformable objects.
• Reusable procedures: Saved programs and procedures reduce execution time in repeated trials, suggesting that runtime-generated solutions can be reused rather than recreated from scratch every time.

Results and limitations

The reported success rates for three block-construction configurations are 100%, 100%, and 80%. Additional material states that assembly, block construction, and die-reorientation configurations each achieved at least eight successes in ten trials. These results indicate that a general-purpose agent can produce workable robot behavior across several real-world settings.

They should not, however, be read as proof of unrestricted robotic generality. The supplied material does not provide the full hardware specification, the complete task scale, a detailed failure taxonomy, or a systematic comparison with specialized policies. The experiments therefore demonstrate feasibility rather than resolving robustness, safety, or broad out-of-distribution generalization.

Why it matters

AGP reframes a robotic policy as a runtime process involving observation, program synthesis, tool use, and correction. This could reduce the need to build a separately trained controller for every new manipulation task and make demonstrations, goal images, or task descriptions more useful as common interfaces.

The same design also exposes important risks. A coding mistake, visual misinterpretation, or unmodeled contact event can immediately become a physical action. More demanding deployments will require safety constraints, action verification, long-horizon evaluations, and reliable recovery from failure. The paper’s main contribution is therefore a path toward more autonomous physical agents, not a claim that the problem of general-purpose robot control has been solved.

Source: Hugging Face Daily Papers