A Two-Person Lab Trained a 2B Text-to-Image Model for About $21K

Introduction

Training a capable text-to-image model is usually associated with large datasets, expensive accelerators, and sizeable engineering teams. Linum, a two-person research lab focused on generative video, has published a different kind of experiment: an approximately 2B-parameter text-to-image model trained with about $21,000 in compute. The weights are released under Apache 2.0, while the team describes the model primarily as a pressure test for its next video architecture rather than a finished product.

Key points

• A pixel-space alternative to VAE pipelines: Conventional latent diffusion systems usually compress images with a variational autoencoder before diffusion operates in the latent space. Pyramid-JiT, or P-JiT, removes that separate VAE and asks the diffusion network to learn a representation suited to both compression and generation.
• More aggressive token reduction: The model works with 32×32-pixel patches and compresses a 512² image into a token budget comparable to what Linum’s earlier setup used for a 256² image. The team reports a roughly fourfold reduction in the attention context window, targeting the quadratic cost of attention.
• Auxiliary readouts during training: Intermediate layers predict clean images at lower resolutions, while a PixelREPA component aligns masked hidden states with DINOv3-L features. These additions are intended to strengthen gradient signals in earlier DiT blocks.
• A staged noise and resolution curriculum: Training begins with a noise distribution concentrated at high noise levels to establish global structure. The schedule is then widened to teach finer detail. The model is trained first at 512² and later continued at 1024².
• A relatively transparent budget: The published accounting lists 8,139 H100 hours at $2.50 per hour, with roughly 610 million samples at 512² followed by 50 million at 1024². Pretraining is listed at about $20,348, bringing the overall run to approximately $21,000.

Why it matters

The interesting claim is not simply that a small lab can reproduce the entire state of the art on a small budget. It is that the usual division between compression and generation may not be necessary. A VAE optimized for reconstruction can impose a ceiling on compression, and better reconstruction does not automatically produce better diffusion samples. P-JiT instead tries to learn a representation shaped by the final generation objective.

That direction becomes more relevant for video. Adding frames greatly increases both the attention context and the memory burden. If a model can converge with fewer samples and operate on fewer tokens without a major quality loss, iterative creative workflows could become more practical. However, this release is still an experimental image model. A training bill and a gallery of samples do not by themselves establish commercial-level quality, prompt adherence, editing control, or video performance.

For open-source researchers, the Apache 2.0 weights and architectural description offer a useful starting point for replication and comparison. The broader lesson is that efficiency may come not only from larger hardware budgets, but also from rethinking the representation, training schedule, and intermediate objectives used by diffusion models.

Source: Hacker News