Realtime-Venus Brings Asynchronous Delegation to Full-Duplex Interaction

Introduction

Most voice assistants still follow a turn-based pattern: the user speaks, the system processes the request, and the assistant responds. This design becomes limiting when the assistant must continuously watch a changing scene, handle an external task, or deal with a user who keeps speaking while the response is being generated. Realtime-Venus approaches the problem as a systems challenge, combining continuous perception, proactive responses, and asynchronous delegation in one interaction loop.

Two models for two real-time settings

The project contains two separately trained 9B models. Realtime-Venus-Omni is built for audio-visual interaction, where the system must connect speech and language with an evolving visual context. Realtime-Venus-Audio targets spoken interaction and focuses on continuous audio understanding and native speech generation. Both models are presented as complete conversational frontends rather than isolated perception components: they integrate perception, conversational control, and speech output.

A central design is a shared causal timeline. User inputs, model outputs, and delegation events are represented within the same temporal sequence. This gives the system a common basis for handling streaming inputs, interruptions, overlapping activity, and context that changes while a conversation is underway.

Keeping conversation alive during background work

Realtime-Venus uses a dual-loop runtime. The foreground loop maintains the live interaction, while Realtime-Venus-Harness runs reasoning, tool execution, or other delegated work in the background. The user can continue interacting while a task is being processed. Once the background result is available, it can be integrated into the ongoing dialogue instead of forcing the conversation to stop and restart.

The architecture changes the role of a real-time assistant. Immediate conversation no longer has to wait for every complex operation to finish, while longer tasks can be handled without abandoning the current context. The supplied material does not specify the exact tools supported by the Harness, deployment requirements, or end-to-end latency, so the design should not be treated as proof of universal production readiness.

Results and broader significance

Among the online models compared in the paper, Realtime-Venus-Omni achieved the highest score on six of eight video benchmarks, including 70.2% on StreamingBench, 64.7% on OVO-Bench, and 81.3% on Daily-Omni. Realtime-Venus-Audio led the compared models on MMAU, MMAU-Pro, Llama Questions, and Speech CMMLU, with reported scores of 78.0%, 63.2%, 83.8%, and 67.8%, respectively.

The training recipe is also notable. It combines offline understanding, proactive full-duplex trajectories, and delegation workflows in post-training. This suggests a shift from assistants that simply answer completed requests toward frontends that remain active and coordinate longer-running work. In practice, however, the quality of that experience will depend on interruption handling, failure recovery, tool reliability, and latency across different devices. Those factors will be important for judging how far the research can translate into robust real-world interaction.

Source: Hugging Face Daily Papers