ENCORE Helps Coding Agents Discover Robot Manipulation Strategies from Demonstrations

Introduction

A robot instruction such as “place the object in the target position” leaves out many details that determine whether the task will work. The robot must infer where to approach, how to grasp, which contacts should happen first, when to release, and what the completed state should look like. Existing systems often address this gap with extensive training data or repeated trial and error. ENCORE proposes a different route: use a few demonstrations as a search prior that a coding agent can inspect before writing a policy.

How ENCORE works

The system does not simply feed demonstrations into a model as additional training examples. Instead, a deterministic builder converts every demonstration into an evidence pack. The pack includes multiview keyframes, gripper events, frame strips, and the complete trajectory. This representation gives the agent several complementary views of both the motion and the task’s implied structure.

A coding agent reads the pack and writes a policy program against a fixed perception and action API. It can run the program during a limited development phase, inspect the resulting rollouts, and revise its code. Once the policy is frozen, it is evaluated in a sealed setting where the success signal is not revealed. The protocol therefore tests whether the agent can infer a reusable strategy from demonstrations rather than patching its behavior in response to evaluation feedback.

Main findings

• On perturbed LIBERO-PRO tasks, the first program succeeds on half of the tasks when demonstrations are available; without demonstrations, it succeeds on one task.
• Frozen programs reach 96.3%, outperforming a prior agentic system using the same language model, which reaches 89.3%.
• On RoboDojo tasks whose instructions do not state the goal, no program succeeds without demonstrations, highlighting the role of examples in supplying missing task semantics.
• ENCORE also runs on a real bimanual robot, learning cube handover and cup inversion from five demonstrations for each task.

Why it matters

The work suggests that demonstrations need not always be converted into a larger training set. They can instead function as structured context that narrows the search space for program synthesis. This is particularly useful in robotics, where language often specifies an intention but omits contact order, motion details, and completion criteria. The evidence-pack design also creates a clearer interface between visual observation, code generation, and physical execution.

The reported results still rely on selected benchmarks, a fixed API, and a limited number of development rollouts. Robustness to poor demonstrations, viewpoint changes, object variation, and more complex environments remains open. Even so, ENCORE frames imitation as an inspectable and iterative program-discovery process, rather than as an opaque end-to-end prediction problem.

Source: Hugging Face Daily Papers