Youtu-Parsing-Omni: One Encoder for Documents, Images, Audio, and Video

Introduction

Document parsing, image understanding, speech recognition, and video analysis are often handled by separate systems with incompatible outputs. Youtu-Parsing-Omni takes a more unified approach: one encoder for every modality and one schema for every parsing task. The 5B-parameter model targets document pages, natural images, charts, flowcharts, geometry figures, audio, and both natural and text-rich video. Its broader goal is to provide a perception layer for agents that can locate, retrieve, and cite evidence.

What the system changes

• A shared omni-modal encoder: Youtu-Omni-Encoder is initialized from a pretrained language model to strengthen text reading, then uses shared weights across image, audio, and video inputs. The representations are organized in a common space-time coordinate system, preserving locations, timestamps, and relationships between modalities.
• Fusion inside perception: Video frames can attend to concurrent audio through a small set of fusion layers. This lets visual content, speech, and acoustic events be aligned while perception is happening, rather than merely concatenated after separate recognition stages.
• One structured output format: OmniSchema places layout elements, bounding boxes, text, LaTeX, tables, charts, flowcharts, OCR, ASR, speakers, acoustic events, actions, and camera motion inside one JSON envelope. The same interface can also carry captions, reports, and narratives.
• Self-evolving post-training: OmniSchema-Aware On-Policy Distillation, or OSAD, trains on the model’s own rollouts. A reference-conditioned copy supplies token-level targets, and the reference is refreshed with the trained weights after each round. In this design, the parser and its supervision co-evolve.

Results and implications

The reported 5B checkpoint reaches 96.96 Overall on OmniDocBench v1.6 and 75.08 average on OmniParsingBench, where it is described as the leading open-weight model among the compared systems. Its advantage is concentrated in audio and video. The model also reports a 95.02 overall chart-parsing score and the highest exact chemical-fingerprint agreement in the comparison.

The important idea is not simply putting several modalities into one model. It is defining a common representation for evidence. For an agent, a result with coordinates, timestamps, reading order, or speaker information is more useful than an ungrounded paragraph: it can support retrieval, citation, verification, and later reasoning. A shared schema can also reduce format-conversion work across multi-model pipelines and make post-processing and evaluation more reusable.

The unified design does not mean every task is equally solved. Performance will still depend on input quality, task prompts, and serving configuration. Long-video compression, precise temporal alignment, and complex geometric relations remain areas that require further validation. Even so, Youtu-Parsing-Omni illustrates a notable shift from recognizing each modality in isolation toward organizing multimodal evidence in a common, agent-ready structure.

Source: Hugging Face Daily Papers