Back to articles

WeMM-Video Brings Audio-Visual Reasoning to an Open Model Family

2 min read

Understanding a video often requires more than recognizing what appears on screen. Speech, background sounds, and visual events can reinforce or contradict one another, making their relationships central to interpreting a scene. Tencent’s WeMM-Video aims to address that challenge with a family of models designed to process text, images, audio, and video together, with particular emphasis on joint audio-visual perception and reasoning. The report describes Chinese-English models in 2B, 4B, 9B, and 35B-A3B configurations.

Key points

  • Two response styles: A hybrid-thinking design supports a non-thinking instruction mode for direct answers and a thinking mode that makes step-by-step reasoning explicit.
  • Large-scale, staged training: The authors describe training on approximately 1.2 trillion multimodal tokens, using progressive pre-training followed by multi-stage post-training. The listed techniques include multi-teacher OPD, MrDPO to address repetition, and specialist reinforcement learning.
  • Broad evaluation: The report says the family was tested on more than 120 benchmarks spanning joint audio-video understanding, general audio, silent video, images, and text.
  • Reported comparisons: The authors say WeMM-Video-35B-A3B sets a new open-source high on omni-modal video evaluations, outperforming Qwen3.5-Omni-Flash and Seed2.0-Mini-0428 while narrowing the gap with Gemini-3.0-Flash. They also report that WeMM-Video-9B surpasses the larger Qwen3-Omni-30B-A3B-Instruct.

The broader significance is the shift from treating video as a sequence of visual frames to interpreting sound and sight as linked evidence. A model that can reason across both modalities could be useful for tasks such as video search and media analysis, although the report summary alone does not establish performance in any particular deployment. It also leaves important details unanswered: no benchmark scores, test protocols, or error analyses are included in the supplied material. The headline comparisons should therefore be read as claims from the authors and checked against the full paper’s evaluation tables. The team says it is releasing the model family along with evaluation and reproducibility materials; assessing practical accessibility will depend on those release details.

Source: Hugging Face Daily Papers

Comments

Checking sign-in status...

Loading comments...

Related articles