One Model, Variable Thinking: ELR Removes the One-Loop Tax

Introduction

Most language models run at a fixed computational depth: the layers used during training are executed once in a standard forward pass. Latent recurrent language models take a different approach by applying a shared block repeatedly to the hidden state. Loop count then becomes a control knob. Fewer loops can reduce latency, while more loops may provide extra computation for a better answer.

The main drawback has been a substantial “one-loop tax.” A recurrent model that runs only once can perform much worse than a non-recurrent model of similar size. If low latency is the priority, that gap makes a conventional model the more practical choice. The paper One Model to Loop Them All introduces Elastic Latent Recurrence, or ELR, as a training recipe intended to close that gap.

The ELR recipe

ELR relies on three design choices rather than merely repeating the same layer block:

• Gated state updates with a learned step size. The model controls how much each pass changes the current hidden state. This helps prevent repeated processing from destabilizing the representation while preserving the quality of the first pass.
• Mixed loop counts within each optimizer update. Training exposes the model to different computational depths instead of optimizing only for one fixed number of loops. The same checkpoint can therefore serve both low-budget and high-budget requests.
• Loop- and budget-aware conditioning. Each pass receives information about its loop index and the available budget. Later passes can then behave as deliberate refinement steps instead of acting like blind repetitions.

According to the paper, the first two choices primarily preserve one-loop accuracy, while the third lowers loss as more loops are added. Mixed-depth training also does not have to begin at the start of pretraining. Introducing it later, with newly added conditioning initialized to zero, reduced the extra training burden and produced lower loss than training with mixed depths from scratch in the reported experiments.

Results and serving considerations

The experiments cover a 4B dense model and a 30B-A3B mixture-of-experts hybrid. Relative to matched non-recurrent baselines, ELR at one loop was 0.52 points lower and 1.04 points higher, respectively, on a ten-task average. Four loops then added 7.25 and 4.75 points. The practical message is not that every request should use four loops, but that one checkpoint can cover a range of latency and quality targets.

MoE models introduce another cost: expert routing must be computed or reused at each pass. The paper reports that reusing the first-loop expert routes on later loops cut mixed-depth update time by 11.7%, with no measurable accuracy change. At inference, keeping only the attention KV cache from the final loop keeps attention KV memory at its one-loop level while retaining most of the benefit from additional passes.

After post-training, the authors present Koi-4B, a single checkpoint with an estimated Artificial Analysis Intelligence Index score of 18 at one loop and 29 at four loops. The paper compares the one-loop estimate with Qwen3.5-4B. These are reported estimates from the study, not a universal guarantee across all evaluations.

Why it matters

ELR reframes part of model selection as budget selection. A service can adjust loop count according to request priority, device load, or latency targets instead of maintaining separate fast and high-quality checkpoints. This could simplify deployment and make elastic inference easier to schedule.

The trade-off remains straightforward: additional loops require additional computation, and their value may vary by task. The reported evidence also comes from specific model sizes and training setups; broader validation is needed for larger models, long-context workloads, and different post-training pipelines. Still, ELR offers a practical principle: variable inference depth works best when it is treated as a capability to optimize during training, not merely as a property added after the fact.

Source: arXiv