When ALiBi Attention Goes Blind: A Hidden Numerical Failure in Long Contexts

Lead

ALiBi, short for Attention with Linear Biases, has been a popular positional encoding choice for Transformer models that need to handle longer contexts. Instead of learning explicit position embeddings, it adds a distance-dependent linear bias to attention scores, encouraging each head to favor nearby tokens while still allowing access to earlier context.

The paper When Attention Goes Blind argues that this simple mechanism has a previously overlooked weakness. Under certain numerical conditions, ALiBi does not merely reduce attention to faraway tokens; it can make those tokens effectively unreachable.

Key points

• The failure is numerical, not just architectural. ALiBi’s linear bias grows with distance. When this bias becomes too large in magnitude, the softmax output for many distant positions can fall below the representable range of floating-point precision and underflow to zero.
• Some attention heads become partially blind. If attention probabilities are exactly zero, the affected head cannot retrieve information from those positions at all. This is more severe than assigning a small but nonzero probability.
• Standard benchmarks may miss the issue. The authors report that the failure can substantially impair token retrieval, especially tasks that require locating a specific item in a long context. At the same time, its effect on conventional decoder benchmarks is comparatively minor, which helps explain why the problem has remained hard to notice.
• The behavior appears in real pretrained models. The paper reports the phenomenon in state-of-the-art pretrained models based on ALiBi, not only in artificial examples.
• Controlled pretraining experiments clarify the cause. Using 148M-parameter decoder models, the authors separate this numerical failure from more general out-of-context degradation.
• Log-scaled distances are the most consistent mitigation. The paper evaluates four training-time mitigation strategies, individually and in combination. Among them, log-scaled distances yield the most consistent improvements on passkey retrieval.

Why it matters

This work reframes part of the long-context challenge as a precision and scaling problem. If distant-token attention has already collapsed to zero, additional training may not be enough for the model to learn reliable retrieval from those positions. The model is not merely preferring nearby content; portions of its attention mechanism have lost numerical access to faraway content.

At the same time, the paper does not conclude that ALiBi should be discarded. Default ALiBi slopes remain a surprisingly strong baseline, particularly on needle-in-a-haystack retrieval. The practical lesson is more nuanced: ALiBi can still be useful, but model builders should test for retrieval failures directly and consider safer distance scaling when training long-context systems.

For practitioners, the takeaway is clear. Language modeling loss and standard decoder evaluations are not sufficient to validate long-context behavior. If a model uses ALiBi, passkey-style retrieval and needle-in-a-haystack tests should be part of the evaluation suite, and training-time mitigations such as log-scaled distances deserve serious consideration.

Source: Hugging Face Daily Papers