Puro-2B Brings 2B-Model Pretraining to Consumer RTX 5090 GPUs

Introduction

Pretraining a language model is often associated with budgets that are inaccessible to small laboratories and independent open-source teams. Puro-2B, introduced by the PACMAN Group at Tsinghua University, investigates a narrower but practical question: how far can a roughly 2B-parameter model be trained from scratch using consumer-grade GPUs and a budget of only a few thousand dollars?

Key points

• Consumer hardware is part of the design. The project uses RTX 5090 GPUs and FP8 precision to reduce the computational and memory burden of training.
• The models are pretrained from scratch. Puro-2B is a collection of runs with different token budgets and recipe variants, reaching up to 1.4 trillion tokens.
• The recipe is a combination of methods. The reported efficiency comes from hardware selection, low-precision training, Hyperball optimization, curriculum model averaging, and a dedicated data recipe rather than from one isolated trick.
• Performance scales with spending. The best reported model costs less than $6,900 in compute and approaches Qwen2.5-1.5B under the authors’ evaluation protocol. A fitted Puro Cost Scaling Law estimates that about $4,400 may be enough to reach the average performance of Qwen2-1.5B in that setup.
• The pipeline is open. Data, code, and model weights are released under Apache 2.0, making the project useful as a reproducible training case study.

Why it matters

Puro-2B does not imply that consumer GPUs can replace large training clusters. Its more important contribution is methodological: it packages hardware choices, numerical precision, optimization, model averaging, and data preparation into an open pretraining workflow. Many open projects publish weights or a training framework, but reproducing the full process can still be difficult when data decisions and stability techniques remain undocumented. Puro-2B aims to make those decisions inspectable.

The cost-scaling analysis is also valuable because it frames pretraining as a set of trade-offs rather than a binary choice. Researchers can consider how token budgets and recipe variants affect quality at different spending levels. At the same time, the estimated $4,400 threshold should be interpreted carefully. It is a fitted result from this model collection and evaluation protocol, not a universal guarantee. Electricity, hardware availability, rental prices, engineering time, data quality, and benchmark coverage can all change the effective cost and the observed performance.

The project also examines how pretraining data curricula influence downstream results after post-training. Access to the complete pipeline allows controlled comparisons that are difficult to perform when only final model weights are available. For the open-source community, this kind of decomposable and reproducible experiment may be more useful in the long term than a single leaderboard number.

Source: Hugging Face Daily Papers