Reflex Teaches Humanoids to Catch Boxes with Egocentric Vision

Introduction: catching is a prediction problem

A person catching a thrown box appears to perform a simple reflex. In reality, the body must estimate the box’s direction and speed, predict where it will arrive, move into position, maintain balance, and close the hands at the right moment. For a humanoid, these requirements combine perception, short-horizon prediction, timing, and whole-body coordination in a single action that unfolds in roughly a second.

A University of Washington team presents Reflex, a framework that studies this problem through the representative task of catching thrown boxes. The robot observes the scene from its own viewpoint using RGB-D cameras rather than relying on motion capture, marked objects, or a controller focused only on the end effectors. The setup is therefore closer to the conditions a mobile humanoid would encounter in an unstructured environment: the target is moving, observations may be delayed or incomplete, and the robot must react while keeping its body stable.

A three-stage training strategy

Instead of learning a direct image-to-joint-action mapping in one step, Reflex separates the problem into three connected stages. A compact representation of object dynamics serves as the interface between them.

• Whole-body catching first. A state-based policy is trained with reinforcement learning. It learns to use the arms, torso, and legs together, rather than treating catching as an isolated hand movement.
• Motion inference next. The policy is then exposed to delayed and partial observations. It learns to infer the dynamics of the moving box and use that estimate to anticipate a future contact point.
• Vision last. An egocentric RGB-D encoder is trained to recover the same dynamics representation directly from visual observations. The whole-body controller is kept fixed during this stage, so visual learning does not have to rediscover the mechanics of the complete task.

This decomposition gives each capability a more direct training signal. It also reduces the burden on visual learning: the encoder only needs to estimate a useful intermediate state, while the established controller handles the conversion from that state to coordinated motion.

Temporal history matters

The experiments emphasize that a single frame is not enough to understand a moving object. A sequence of observations provides clues about velocity and trajectory, allowing the robot to distinguish where the box is from where it is going. When temporal visual history is removed, success falls by 35.6 percentage points for centered throws and by 39.8 points for laterally offset throws. The result highlights a broader issue in reactive robotics: visual perception must preserve motion information, not merely recognize objects.

In simulation, the egocentric RGB-D system achieves an 85.7% catch success rate. With privileged observations of the box, success reaches 87.9%. The relatively small gap suggests that the learned visual encoder can recover most of the dynamics information needed by the controller, although the visual problem remains more difficult than directly exposing the target state.

From simulation to a Unitree G1

Reflex was also evaluated on a real Unitree G1 humanoid catching boxes thrown by a person. At a distance of 2.5 meters, the robot succeeds 65% of the time; at 5 meters, success falls to 40%. Human catchers achieve higher success at both distances, and the robot reaches 76% and 59% of human performance respectively.

These results are meaningful because they move beyond an idealized simulation, but they also show the remaining gap. Longer distances amplify perception delay, trajectory uncertainty, and the amount of body motion required before contact. The reported experiments focus on a box-catching scenario, so performance across different object shapes, weights, throwing styles, lighting conditions, and occlusions remains an open question.

Why it matters

The main contribution of Reflex is a practical recipe for reactive whole-body control. First teach the body how to act, then teach the system to infer target dynamics, and only afterward connect egocentric vision to that intermediate representation. The same pattern could support tasks such as avoiding moving obstacles, receiving objects during collaborative work, or responding to dynamic handoffs.

Reflex does not make humanoids universally reactive, but it shows a path from passive visual recognition to action grounded in short-term prediction. For embodied AI, that transition may be as important as improving object detection itself.

Source: Hugging Face Daily Papers