Embodied First Exam: A Robotics Entrance Test for Coding Agents

Introduction

A new direction in robotics is emerging alongside conventional visuomotor policy learning: coding agents can inspect a task, write software, and sometimes train a policy before handing control to a robot. This raises a basic evaluation question. Can a general-purpose coding agent reliably operate different robots across diverse tasks and under the constraints of the physical world?

Most existing robotics benchmarks were designed around a fixed visuomotor policy. They are not necessarily built to measure whether an agent can choose a control strategy, interact with an environment, recover from failure, or make engineering decisions during execution. Embodied First Exam is proposed as an agent-native answer to that gap.

Key points

• The agent chooses the control paradigm. It may directly issue control commands, write code to control the robot, train a policy, or combine these approaches. The benchmark therefore evaluates the decision about how to control, not only the final trajectory.
• The task coverage is broad. The framework includes 1,156 tasks across 35 suites and 47 robots. Nine new suites move beyond ordinary pick-and-place scenarios into spatial puzzles, dexterous manipulation, multi-robot interaction, dynamic whole-body control, and sim-to-sim engineering.
• Three settings separate different capabilities. In the Standard Setting, the agent receives sensor observations and, by default, cannot reset the environment. The Realtime variant keeps the environment running while the agent reasons. The Privileged Setting exposes the full simulator state and allows unlimited resets, producing an approximate upper bound useful for training-oriented analysis.
• The current performance gap is substantial. On 456 core tasks, the strongest of eight evaluated agents, GPT-6 Astra in Codex, achieved a 34% success rate in the Standard Setting. The result suggests that code generation alone does not translate into dependable physical control.
• Failures are treated as data. The benchmark labels the stage at which an attempt fails and compares failure patterns across agents, settings, and stage categories. This can help distinguish problems in perception, planning, code generation, policy learning, or execution.

Why it matters

The main contribution is not simply the number of tasks. It is the shift in what counts as the unit of evaluation. A conventional benchmark may ask whether a pre-trained controller can complete an action. Embodied First Exam asks whether an agent can understand the task, select an appropriate control method, use feedback, and complete a closed-loop interaction under limited information and imperfect execution.

The distinction between the three settings is particularly useful. Full simulator access and unlimited resets can help an agent search for an oracle-like solution, but they do not represent ordinary operation in the physical world. The Standard Setting introduces the more consequential constraints: limited observations and, by default, no easy restart. The Realtime variant adds another source of difficulty by allowing the world to continue changing while the agent deliberates.

The 34% result should therefore be read as a capability baseline rather than a final verdict on coding agents. It shows that there is substantial room for improvement before these systems can be considered reliable general-purpose robot operators. At the same time, the benchmark creates a common language for diagnosing that gap. Future systems can be compared not only by whether they succeed, but also by whether they are better at choosing a control route, recovering from mistakes, and maintaining performance across robot embodiments.

The framework remains a simulation-based first exam, not a certification for safe deployment in the real world. Its value is precisely in making the pre-deployment problem more explicit: before a coding agent enters a physical environment, researchers need to know which part of the control pipeline still fails.

Source: Hugging Face Daily Papers