Back to articles

Why Probing and Steering Prefer Different Directions in Language Models

2 min read

A direction in a language model’s hidden state can serve two purposes: it can indicate whether a concept is present, or it can be added to the state to push the model toward a behavior. Because the same kinds of vectors are often used for both, it is tempting to assume that a direction that reads a concept well will also control it well. This study tests that assumption—and finds the two goals frequently diverge.

The authors examine three instruction-tuned models from different families, focusing on truthfulness and sentiment. They define a continuous family of directions, spanning a covariance-adjusted discriminant at one end and the difference between class means at the other. They then assess both probing performance and additive steering, comparing steering effects at a fixed level of side effects.

The results point to three main takeaways:

  • In every model, the best probe direction is not the best steering direction. Steering consistently favors directions closer to the mean difference, suggesting that making a concept easy to distinguish is not the same as finding an effective intervention.
  • The separation holds across 59 of 60 layer–model–concept combinations in a layer sweep. It also remains in all six model–concept combinations when the probing and steering texts are matched, and persists with a rotation-based steering operator.
  • The precise steering optimum is less consistent. For sentiment, an intermediate direction produced a 27–56% stronger effect than the mean difference in some comparisons. That advantage shrank or disappeared with matched texts and varied with layer and operator.

The study also reports an asymmetry in cross-model transfer. A probe direction transferred through a linear map fitted on neutral text retained 81–91% of the target model’s above-chance probing performance. The transferred mean-difference direction retained less of its steering effect. The available source excerpt cuts off before giving further details, so the size of that steering-transfer loss cannot be specified here.

The broader lesson is methodological: a readable feature is not automatically a useful control handle. Probing asks whether information can be extracted; steering asks whether an intervention can cause a desired change while limiting side effects. Those objectives can favor different directions, and results can depend on the layer, text set, and steering operator. Researchers using activation vectors for model editing or behavioral analysis should therefore evaluate each intended use separately rather than treating probe quality as a proxy for control.

Hugging Face Daily Papers

Comments

Checking sign-in status...

Loading comments...

Related articles