DeepLoop: Stabilizing Depth Scaling in Looped Transformers

Introduction

Looped Transformers offer a parameter-efficient way to deepen sequential computation. Instead of storing a separate set of parameters for every layer, a compact stack of physical Transformer blocks is applied repeatedly. This increases the model’s unrolled depth without increasing the number of stored parameters, making the idea attractive for language models that aim to trade additional computation for stronger reasoning or representation capacity.

DeepLoop focuses on a less obvious consequence of this design. A looped Transformer is not simply a standard Transformer with more layers. When the same block is visited multiple times, its parameters receive updates aggregated from several visits and are then read again by those same visits in the next linearized forward pass. That tied-depth behavior changes the residual-scaling problem.

Key points

• Parameter reuse changes the training dynamics. In an untied Transformer, each residual branch has its own parameter update. In a looped Transformer, the same update is shared across repeated visits, creating coupling between visits.
• The paper formalizes this effect. The authors derive a first-order perturbation bound controlled by a visit-alignment coefficient, κ_R, which captures how aligned repeated visits are.
• DeepNorm is recovered only in a decorrelated regime. When visits decorrelate, the bound recovers the familiar DeepNorm exponent. But under a conservative aligned regime, as the loop count grows while physical depth remains fixed, the exponent must increase from 1/4 to 1/2.
• DeepLoop keeps the architecture simple. Rather than introducing a new Transformer block, it retains the Post-LN DeepNorm architecture and sets α=(2N)^{1/2} and β=(8N)^{-1/2} for unrolled depth N.
• Empirical results are conditional on actual recurrence. On GPT-style looped language models at GPT-2 small and GPT-2 medium scale, DeepLoop is neutral when no physical block is revisited, but improves validation loss and downstream accuracy once recurrent depth is activated.

Why it matters

The main contribution of DeepLoop is a shift in how depth scaling should be viewed for recurrent Transformer designs. Counting the nominal number of layers is not enough when the same parameters are visited repeatedly. Stability depends on the pattern of parameter visits and on how gradients and updates align across those visits.

This is relevant to a broader line of work exploring parameter-efficient LLMs, recurrent computation, and test-time compute scaling. If models are expected to gain capability by looping over a compact set of blocks, their initialization and residual scaling rules must reflect that recurrence. Otherwise, extra computation may fail to translate into stable or useful depth.

The current evidence is limited to GPT-style looped language models at GPT-2 small and medium scales, so it remains to be seen how DeepLoop behaves in larger systems or with different training recipes. Still, the paper provides a concrete lesson: stable recurrent depth requires scaling rules designed for parameter visits, not just for layer count.

Source: Hugging Face Daily Papers