Loopie Reopens the Case for Looped Transformers

Introduction

Most Transformer scaling stories follow a familiar formula: add more parameters, add more data, and spend more compute. Loopie, introduced in the paper “Loop the Loopies!”, explores a different route. Instead of treating depth and capacity as something that must always come from larger static models, it revisits the idea of reusing Transformer computation through loops.

Key points

• Model design: Loopie is presented as a family of looped Transformer models built with Mixture-of-Experts (MoE). The series contains two models: one with 20B total parameters and 2B active parameters, and another with 6B total parameters and 0.6B active parameters.
• The core challenge: Looped Transformers have faced a practical scaling problem. When pre-training compute grows by a factor of N, increasing parameter count by N has usually performed better than running the same model through N loops.
• What the paper claims: The authors argue that Loopie addresses this gap. Through extensive ablation studies, including comparisons against a vanilla 30B-A3B model, they report that Loopie substantially outperforms standard Transformer baselines trained with the same compute budget.
• Post-training for reasoning: Beyond architecture, the paper highlights a post-training pipeline designed to strengthen reasoning. The abstract states that Loopie reaches gold-medal-level performance at the 2025 IMO and IPhO without using tools.

Why it matters

Loopie is interesting because it shifts the scaling discussion from “how large can the model be?” to “how effectively can computation be reused?” MoE already offers a way to separate total capacity from active compute. Looped Transformers add another dimension: the possibility that repeated passes through shared components can produce deeper computation without simply multiplying parameters.

If the reported results hold across broader evaluations, Loopie could point toward a more compute-efficient scaling path for large language models. That would be especially relevant as training costs continue to rise and researchers look for alternatives to brute-force parameter growth.

There are still important caveats. The available material is mainly the abstract and paper listing, so the details of the architecture, data, training recipe, evaluation setup, and post-training pipeline need to be examined in the full paper. Claims about equal-compute comparisons and competition-level reasoning are particularly sensitive to methodology.

Even so, Loopie is a notable signal: looped computation, sparse expert activation, and targeted post-training may become a meaningful combination in the next phase of LLM efficiency research.

Source: Hugging Face Daily Papers