Sparse Autoencoders Reveal Fine-Grained Specialization in MoE Experts

Introduction

Mixture-of-Experts models expand model capacity by routing each token to only a small subset of available experts. Yet routing alone does not answer a basic interpretability question: what does an individual expert actually do? Counting how often an expert is selected on a corpus labeled by broad domains can be useful, but it fixes the taxonomy in advance and measures selection rather than contribution.

A new study takes a feature-space approach. It trains sparse autoencoders (SAEs) on the outputs of MoE layers, then uses the resulting features to identify which experts are primarily responsible for particular patterns in the model’s representation.

A contribution-based method

The experiments examine layers 4, 30, and 46 of Qwen3-30B-A3B. For each layer, the authors train a TopK SAE with 2^20 latent features and retain the top 100 active features for each input. The analysis aggregates evidence from roughly 1.5 billion tokens.

The key technical observation is that an MoE output is a router-weighted sum of expert outputs. Because the SAE encoder is affine, the pre-activation of every latent can be decomposed exactly into contributions from the participating experts. Across the corpus, this makes it possible to identify a dominant set of latents for each expert: features whose activation is principally driven by that expert.

This avoids imposing categories such as “code” or “mathematics” before the analysis begins. Instead, the structure of the learned features determines where expert boundaries appear.

Main findings

• Contribution is more concentrated than gating. An expert may be selected alongside others without being the main source of the resulting feature. Looking at contribution therefore produces a sharper picture of specialization than counting router decisions.
• Most features have one or two dominators. The majority of SAE latents are principally driven by only a small number of experts. Dominant feature sets are largely private, although a limited number of expert pairs show notable overlap.
• Specialization exists below broad domains. Automatically interpreting the dominant sets reveals separate roles inside familiar categories. The study identifies distinct experts associated with Java and frontend web code, as well as separate patterns related to geometry, arithmetic, and polynomials.
• Expert allocation is uneven. Grouping the 128 experts in layer 30 yields 11 higher-level domains of very different sizes. Some groups are not conventional subject areas at all, but patterns such as text form, language, entities, and identifiers.

Why it matters

The study suggests that MoE organization is not simply a one-expert-per-domain arrangement. Instead, experts appear to participate in a layered division of labor built from many fine-grained features. This distinction matters for interpretability, routing design, expert pruning, and attempts to specialize or adapt only parts of a large model.

There are also important limits. The conclusions are based on SAE decompositions and automated interpretation of feature sets across three layers. They show structured associations, but do not by themselves establish that removing an expert will causally eliminate the corresponding capability. Testing more architectures, layers, and data distributions—and intervening on experts directly—will be needed to establish how stable these roles are.

The authors release the three SAEs, their code, and an interactive explorer covering all 128 experts in layer 30, making the analysis available for further inspection.

Hugging Face Daily Papers