SAEScientist-Bench Tests Whether AI Agents Can Do Mechanistic Interpretability Research

Introduction

Automating model training is only one part of autonomous AI research. If an agent is expected to improve models reliably, it must also inspect what a model has learned, identify internal mechanisms related to a target behavior, and check whether an intervention has the intended effect. SAEScientist-Bench addresses this missing evaluation layer by asking whether agents can conduct sparse autoencoder research with limited human guidance.

Turning feature search into an experiment

Sparse autoencoders, or SAEs, decompose model activations into a large collection of relatively sparse features. These features can provide a practical interface for inspecting concepts inside a model and testing whether selected features can be used to influence behavior. The benchmark does not reduce the task to looking up a feature ID. Given a target concept, an agent must design contrastive probes and navigate a Gemma Scope dictionary containing more than 131,000 features.

The experiments use Gemma-2-9B-IT and compare agent discoveries with curated expert reference features anchored on Neuronpedia. Evaluation covers three connected questions:

• Feature discovery: Can the agent identify a feature associated with the target concept?
• Concept selectivity: Does the feature distinguish target texts from contrastive controls rather than responding to superficial correlations?
• Causal steering: Does manipulating the feature actually change model generation in the intended direction?

Across 10 agent configurations and 20 tasks, the results show genuine but uneven capability. Frontier agents find promising candidates and perform relatively well at separating target concepts from controls. On that dimension, they approach expert-level behavior. Their performance is substantially weaker when the task requires causal steering of generated text. In practical terms, agents are better at answering “which feature appears relevant?” than “will changing this feature reliably change the model’s behavior?”

The bottleneck is interpretation, not only search

The deeper analysis is especially important. Agents are not incapable of experimental reasoning altogether: they can construct contrasts that help eliminate spurious candidates. However, they frequently misinterpret measurements such as activation rankings, selectivity results, or changes observed after intervention. This indicates that the challenge is not simply the size of the feature space. It also lies in connecting experimental design, metric interpretation, and causal reasoning into a coherent conclusion.

Why it matters

SAEScientist-Bench makes experimental understanding of models a more concrete capability to measure. For autonomous AI R&D, training automation answers how a model can be changed; mechanistic evaluation asks whether an agent understands what changed and why. That distinction is important for monitoring, auditing, and alignment work.

The findings also argue against treating feature retrieval as equivalent to explanation. A feature that correlates with a concept may not be causally effective, and a successful intervention still requires careful interpretation. Future systems will likely need stronger verification loops, better separation of correlation from causation, and more reliable checks on the conclusions agents draw from experiments.

Source: Hugging Face Daily Papers