MOSS-VL Makes Seeing While Speaking a Core VLM Capability

Introduction

Most vision-language models follow an offline pattern: receive a complete image or video segment, then produce an answer. A real-time assistant faces a different constraint. Frames continue to arrive while the model is already responding, so it must observe, reason, and generate without waiting for the entire visual context. The MOSS-VL Technical Report treats this “perceiving while speaking” problem as a first-class design target.

What the system changes

• Vision enters through gated cross-attention. Instead of placing visual tokens directly inside the sequence being decoded, the language decoder accesses visual representations through gated cross-attention. This allows incoming frames to remain available during generation and keeps visual context outside the decoded token sequence.
• Interaction behavior is explicitly supervised. The team synthesizes an interaction corpus that teaches the model when to speak proactively, when to stay silent, and when to revise a previous response after new evidence appears. These behaviors are central to a useful streaming assistant but are not captured by ordinary image-question-answering data.
• Real-time adaptation is concentrated at the end of training. MOSS-VL first builds on a strong offline vision-language foundation, then concentrates real-time-specific training in a lightweight final stage. This curriculum is intended to add streaming behavior without requiring the entire model to be trained from scratch for that setting.

Reported results

The report says MOSS-VL-Instruct remains competitive with models at a comparable scale in offline evaluations and leads temporal-reasoning video sets. MOSS-VL-Realtime achieves the best average among open-source streaming models on three of four streaming benchmarks and ranks second on the remaining benchmark. It also leads all three subsets focused specifically on proactive behavior. On OmniMMI Proactive Alerting, it scores 66.0, compared with 37.5 for the strongest baseline cited in the report.

MOSS-VL has 11.3 billion parameters. Because visual tokens sit outside the decoded sequence, its time-to-first-token advantage over the same-backbone Qwen3-VL-8B grows from 2.8x to 5.1x as visual context becomes larger. The result highlights an important systems point: streaming latency depends not only on parameter count, but also on how visual information is connected to generation.

Why it matters

MOSS-VL offers a clear decomposition of real-time multimodal interaction. The architecture supports continuous visual access, the data teaches timing and correction behavior, and the curriculum limits the amount of specialized training required. Together, these choices provide an open recipe for video assistants, proactive alerts, and other continuously observed environments.

The available material does not include every benchmark setting, hardware configuration, or detailed error analysis. The reported scores therefore establish a promising direction rather than a complete deployment picture. Long-running stability, throughput, and the reliability of proactive interruptions still require broader testing. The release of five checkpoints, the training curriculum, and real-time inference code should make those follow-up comparisons easier.

Source: Hugging Face Daily Papers