Motion-Omni Makes Dialogue Models Decide What to Say and How to Move

Introduction

A conversational avatar needs to do more than turn text into speech. Its facial expression, hand gestures, posture, and lower-body motion should follow the meaning and rhythm of what it says. Most existing systems handle this with a cascade: a dialogue model first produces an audio response, then a separate co-speech motion model animates the avatar from that completed audio. The design is practical, but it requires another full inference pass and prevents the speech and motion modules from being optimized together.

Motion-Omni, proposed by a Peking University team, takes a different route. It connects the language model, speech generator, and motion generator so that speech and motion are produced from the hidden states used to generate the response. The output includes explicit facial expression as well as hand, upper-body, and lower-body motion.

Key points

• From cascading to joint generation: Motion is not produced only after the waveform is complete. It is derived directly from the internal representations that also support speech generation, giving the model a shared basis for semantic and temporal coordination.
• Co-training matters: The study reports that freezing the speech pathway leaves motion poorly aligned with the audio. Alignment improves when the LLM, Speech Generator, and Motion Generator are adapted together under both speech and motion objectives.
• Scalable pseudo-labeling: A replaceable motion teacher labels consistent-voice speech responses, producing 422,856 quality-ranked pairs covering 1,402 hours. This offers a way to expand training resources without manually annotating every gesture and body movement.
• A unified benchmark: The authors release SwDA-500 and propose a public protocol for stochastic, open-ended full-body spoken dialogue. It combines common rendering, automatic metrics, human ratings, and latency measurements, while matching audio across motion systems for fairer comparisons.
• Real-time performance: With a Qwen2.5-7B-Instruct backbone, Motion-Omni-Q7 stays within 2% of the teacher cascade on reference-free motion metrics. It reaches a real-time factor of 0.78, which is faster than real-time playback, and reports a 2.62% word error rate, the lowest among the compared omni-modal systems.

Why it matters

The contribution is not simply the addition of more output heads. Motion-Omni suggests that speech and bodily expression can be coordinated through a common internal representation instead of being treated as two disconnected post-processing tasks. That design may reduce inference overhead and create a more direct link between language, prosody, emotion, and gesture.

The results should still be read within the paper’s setup. Pseudo-label quality, the coverage of motion styles, and robustness across languages, characters, and interaction contexts require further testing. The released dataset and evaluation protocol are therefore important as much for future verification as for the reported numbers themselves.

If the approach scales, real-time avatars, game characters, digital assistants, and embodied interfaces may increasingly generate speech and movement as one multimodal decision rather than as sequential modules.

Source: Hugging Face Daily Papers