Gemini Robotics ER 2 brings video understanding and orchestration to robots

Lead

Google DeepMind has launched Gemini Robotics ER 2, a new embodied reasoning model intended to serve as a high-level brain for robots. Rather than directly replacing low-level controllers, it sits above them: it interprets human instructions, observes video streams, plans multi-step tasks, and delegates execution to tools such as VLA models, navigation APIs, manipulator controls, Google Search, or user-defined functions.

The key shift is not simply better perception. ER 2 is designed to reason while a robot is already acting, monitor whether a task is progressing correctly, and decide when to retry, switch steps, or involve another robot.

Key points

• Tool orchestration for physical agents: Developers can expose low-level capabilities as tools. Gemini Robotics ER 2 then selects and sequences those tools to complete more complex workflows.
• Lower-latency execution loop: The model integrates with the Gemini Live API through a bidirectional streaming endpoint, aiming to reduce the awkward “stop and think” pauses that make robots less fluid in the real world.
• Progress tracking from video: ER 2 watches continuous video feeds to estimate task completion. In DeepMind’s evaluation setup, frames were assigned to five progress bands from 0–20% through 80–100%, with ER 2 reaching 57.4% accuracy on progress classification.
• Moment finding: The model also identifies the precise moment when a critical event occurs, such as when to stop pouring into a cup. DeepMind reports 91.3% accuracy and a 0.96-second mean absolute distance on moment-finding tasks.
• Multi-robot collaboration: ER 2 can coordinate different robots through shared semantic understanding, enabling handoffs between machines with different strengths. DeepMind cites a collaboration example involving Apptronik’s Apollo 2 and Franka F3 Duo.
• Spatial reasoning and safety improvements: The release also emphasizes video-based success or failure detection, broader instrument reading, spatial VQA, safety instruction following, and human proximity awareness.

Why it matters

Real-world robotics often fails in the gaps between steps. A robot may grasp an object but drop it, pour too much liquid, misalign a tool, or move on before a task is actually complete. By combining video understanding, temporal reasoning, and tool calling, Gemini Robotics ER 2 tries to make robots better supervisors of their own execution.

The multi-robot angle is equally important. Instead of assuming one general-purpose robot can handle every environment, ER 2 points toward systems where wheeled robots, humanoids, and robotic arms cooperate according to their strengths. Still, most of the available evidence comes from DeepMind’s own demos and evaluations. Broader deployment will need to test reliability, latency, cost, and safety boundaries in messier real-world settings.

Source: Google DeepMind