Gander Brings Continuous Multimodal Interaction and Agentic Reasoning Into One Framework

Introduction

Many voice assistants still follow a simple pattern: the user finishes speaking, the system processes the request, and a response is generated. Agent systems add another layer of complexity by connecting perception, planning, tool use, and output generation across separate components. The paper Omni Interaction Agent Technical Report presents Gander as an attempt to combine these capabilities in a single end-to-end framework.

Rather than treating interaction as a sequence of isolated turns, Gander is designed to continuously receive video, speech, and text. The user may interrupt the model at any point, while the model can provide interim feedback or ask a follow-up question before the larger task is complete.

Key points

• Streaming multimodal input: Gander is built for continuous streams rather than complete, neatly separated requests. This is intended to support more natural full-duplex conversation.
• Cerebellum-Brain collaboration: The Cerebellum is responsible for low-latency interaction and omni-modal conversation. The Brain handles complex reasoning and higher-level agentic work. Tool calling and an agent orchestration runtime connect the two components.
• Streaming Thinker-Talker design: Within the Cerebellum, user inputs and model outputs are flattened into an ordered token stream at chunk level. The goal is to provide a common representation for incremental processing and faster conversational responses.
• Four evaluation dimensions: The paper describes evaluations covering conversational ability, omni understanding, interactive capability, and agentic intelligence. The available material mentions internal human evaluations, but does not provide detailed datasets, scores, or baselines.

Why the architecture matters

The most interesting aspect of Gander is not simply the addition of vision or speech. It is the attempt to redesign the timing of interaction. In a conventional pipeline, speech recognition, reasoning, tool execution, and speech synthesis often happen in sequence. Interruptions and follow-up questions can therefore introduce delays or require state to be passed between loosely connected modules.

Gander’s division of labor addresses this tension directly. The Cerebellum can maintain the rhythm of an ongoing conversation, while the Brain works on planning, reasoning, and tool use. In principle, a difficult workflow does not have to block every immediate conversational response. A user could receive progress updates while the system continues working on a longer task.

This design could be useful in collaborative assistants, workflow automation, real-time visual understanding, and situations where a task requires repeated clarification. It also points toward a broader shift: agent systems may need to be judged not only by final task success, but also by how well they manage timing, interruptions, uncertainty, and user expectations.

There are still important open questions. The supplied information does not explain how the two components synchronize state, when the model should proactively speak, how interruptions affect tool calls, or what latency and compute costs the design introduces. Without those details, Gander should be viewed as a substantial architectural proposal rather than a fully quantified proof of superiority.

Source: Hugging Face Daily Papers