A New Look at Multimodal Pretraining: Knowledge Flow, Synergy, and Early Unification

Lead

Multimodal foundation models are increasingly moving toward native, unified pretraining. But the field still lacks a clear explanation of how language and vision actually interact during training. This Meta study tries to fill that gap by treating multimodal pretraining almost like a system with measurable dynamics: what flows where, when modalities cooperate, and when they get in each other’s way.

Key takeaways

• Knowledge flow is directional and uneven: language, visual understanding, and visual generation do not transfer benefits in the same way.
• Synergy depends on complexity: on harder data and tasks, modalities are more likely to complement each other; on simpler setups, they can compete for capacity.
• Architecture matters: shared attention and shared normalization, combined with modality-specific feed-forward layers, are highlighted as a strong design choice.
• Early unification wins: training modalities together from the beginning is more effective than aligning them later or training them sequentially.
• Delayed vision can cause “vision laziness”: if image input arrives too late, the model may lean too much on language priors.
• Efficient recipes are possible: the authors report strong generative results with only about 5% of the compute budget.

Why it matters

The main contribution here is a more principled mental model for multimodal training. Instead of assuming that “more modalities” automatically means better performance, the paper shows that training outcomes depend on how information moves across modalities, how complex the data is, and whether the architecture encourages cooperation rather than competition.

For practitioners, the message is straightforward: if you want vision to matter, integrate it early, design for shared representations where it helps, and do not assume late fusion will recover the same behavior. The study also claims these insights scale, having been validated with multiple 13.5B MoE models trained on 2T tokens.

Hugging Face Daily Papers