MAEP Prunes MoE Experts by Modality to Fit Multimodal Models on 40GB GPUs

Introduction

Mixture-of-Experts (MoE) architectures expand model capacity by routing each token to only a small subset of experts. This keeps the active computation sparse, but it does not eliminate the memory cost of storing the experts. In multimodal large language models, expert weights can account for more than 90% of the model size. The material cites InternVL3.5-30B-A3B and Qwen3-VL-30B-A3B as examples: their BF16 expert weights occupy roughly 54GB out of an overall footprint of about 58GB.

This creates a practical deployment problem. Removing experts can make the model easier to fit on consumer or single-GPU hardware, but a pruning rule that ignores modality may damage either visual understanding or language generation.

The central idea: score experts by modality

Many existing pruning approaches aggregate importance across all tokens. That assumes an expert has the same role for images and text. MAEP, short for Modality-Aware Expert Pruning, challenges this assumption. It estimates expert importance separately for vision and language tokens during a single forward pass, then ranks experts globally across layers.

Its design has three notable elements:

• Delta-H Output (ΔH): This metric estimates how much the MoE output changes when an expert is removed. It is intended to account for redistribution effects in the routing process, rather than relying only on a gate score or raw activation count.
• Low-attention vision tokens: MAEP uses vision tokens receiving low attention as a signal for the vision-side pruning decision. According to the material, the resulting expert-importance ranking remains stable across datasets, making it less dependent on a particular evaluation set.
• Separate preservation and pruning signals: Text-side importance is used to protect language behavior, while vision-side importance helps identify experts that can be pruned. The final ranking is performed across layers instead of independently within each layer.

Results and implications

The reported experiments cover three MoE-based multimodal language models and both multimodal and text-only benchmarks. Compared with modality-agnostic baselines, MAEP performs better on most multimodal benchmarks while remaining competitive on text tasks. With approximately half of the experts removed, it reduces the BF16 expert-weight footprint by about 27GB, or roughly 47% of total memory. The material says this allows the full weight footprint of a model around 58GB to fit on one 40GB GPU.

The broader contribution is a shift in how multimodal compression is framed. Experts should not necessarily be treated as interchangeable capacity blocks: their utility can depend on whether the input is visual or linguistic. A training-free procedure also lowers the cost of adapting an existing model, since it does not require an additional optimization stage.

There are still open practical questions. The supplied material does not include the full benchmark scores, the effect on inference latency, or performance curves at different pruning ratios. Those details will be important for deciding whether MAEP is suitable for a particular model and deployment workload.

Hugging Face Daily Papers