Ex-Omni-2D: A Multimodal Dialogue Framework with Visual Presence

Introduction

Most omni-modal dialogue systems can already process images, audio, and text, and many can generate spoken answers. But the response often remains disembodied: the system speaks without appearing. Ex-Omni-2D targets this limitation by designing a dialogue framework that can respond with text, personalized speech, and expressive video together.

Key points

• Three outputs in one response: text, personalized speech, and reference-conditioned video are generated in a coordinated way.
• Plan before generation: given a multimodal query, a reference image, and reference audio, the model first predicts a structured Visual Thought Plan (VTP) that captures scene, emotion, and motion.
• Shared acoustic-temporal interface: native multi-codebook speech units serve as a common interface. They can be decoded into speech and aligned online with video frames.
• Less dependence on large joint supervision: because speech, dialogue, and avatar-video data can all contribute to learning, the system does not require massive query-text-speech-video supervision.
• Teacher-student streaming strategy: a full-sequence Video Generator is used as the teacher, then distilled into a few-step block-causal Streaming Student. Its Prefix Streaming mechanism carries a cleaner latent across chunks to reduce late-stage quality decay.

Why it matters

The main contribution is not only technical integration, but also a shift in what a dialogue model can feel like. Instead of returning a purely verbal answer, Ex-Omni-2D aims for a response with presence: expression, motion, and synchronization between speech and video. That makes it especially relevant for avatars, interactive assistants, and embodied conversational systems.

The efficiency side is equally important. The paper reports an end-to-end real-time operating point for the full four-GPU pipeline under four-step inference. This suggests the approach is not just a research concept, but an attempt to balance quality and deployment practicality.

If this direction continues to mature, multimodal assistants may move from being merely responsive to being visually embodied participants in conversation.

Source: Hugging Face Daily Papers