HexaShell Tests How Far VLM Coding Agents Can Control Robots

Introduction

Vision-language models can now interpret images, generate programs, and operate software through tools. Robotics adds a much harder layer: the agent must connect visual observations to spatial coordinates, motion, and physical contact. HexaShell asks how far a coding agent can go when its “computer” is replaced by a two-arm robot.

The project is intentionally built around a thin harness. Instead of embedding a large collection of task-specific policies in the robotics system, the harness mainly converts the agent’s commands into robot motion. The agent controls the robot through one command-line interface and receives observations from three RGB cameras together with their calibration information.

An agent that turns failures into code

HexaShell treats failure as part of the programming loop. After an unsuccessful attempt, the agent reads the execution log, diagnoses what went wrong, and writes or edits a small program known as a tool. The tool can then be reused on later attempts. In this sense, the system resembles a coding agent that fixes a failing test and keeps the fix in the codebase rather than relying only on an ephemeral chain of thought.

The tools are constrained to information the robot can observe. They use geometric reasoning to estimate object locations and relative positions:

• Two cameras observing the same point provide two rays whose intersection estimates the point’s location.
• If a point is known to lie on a surface such as the table or the top of a block, the system intersects the camera ray with that plane.
• When the head camera cannot resolve the scene, the robot moves a wrist camera to obtain another view.

This division of labor is important. The model is not handed a hidden state representation or a prebuilt task policy. It must use observations, geometry, logs, and code to construct the missing procedure.

Evaluation and what the score means

For each task, the researchers build tools using only ten layouts from one seed. They then freeze those tools and evaluate them on ten new layouts from the same seed. This setup tests whether the generated procedures generalize beyond the scenes that motivated their creation, while keeping the evaluation protocol tied to the RoboDojo benchmark.

Across all 42 RoboDojo tasks, HexaShell reports a 39.7% success rate and a progress score of 48.0, averaged according to the leaderboard procedure. The paper describes this as a new state of the art on RoboDojo. The system uses Codex with GPT-6-Astra at medium reasoning effort. These results should be read as benchmark evidence, not as proof of general-purpose robotic competence.

Why the approach matters

The most interesting contribution is the treatment of code as an embodied agent’s memory. A failed action can produce an explicit, inspectable, and reusable artifact. That makes improvement more modular than repeatedly asking a model to improvise an entire trajectory from scratch. It also creates a clearer interface for debugging: researchers can inspect the generated tools, the logs, and the geometric assumptions.

The paper’s central diagnosis is equally significant. Its reported limit is contact rather than reasoning. A model may identify a target and produce a plausible sequence, yet still fail when small localization errors, occlusion, or the dynamics of contact affect grasping and pushing. For robot coding agents, improving physical interaction and robustness may therefore matter more than simply extending prompts or increasing reasoning budgets.

Source: Hugging Face Daily Papers