Halcyon Questions Whether Language Pretraining Is Needed for Text-to-Image Models

Introduction

Many modern text-to-image systems follow a familiar recipe: pretrain a language model on internet-scale text, then connect its language capabilities to a visual generation system. A new paper from Stanford examines whether that recipe is actually necessary when the target task is text-conditioned image generation.

The paper introduces Halcyon, a family of multimodal autoregressive models trained entirely from scratch. Rather than starting from a pretrained large language model, Halcyon is trained on GPIC, a dataset containing 100 million image-text pairs.

What Halcyon does

Halcyon uses an early-fusion multimodal architecture. Image and text representations enter a shared modeling process relatively early, instead of being developed separately and combined only near the end. The design is intended to let the model learn the relationship between visual tokens and language directly from paired data.

The paper highlights two comparisons:

• Halcyon-0.52B achieves a reported gFD of 48.0, compared with 63.3 for LlamaGen-XL/8. The authors also report 18.2 times less pretraining compute for Halcyon.
• Halcyon-1.6B reports a gFD of 35.3, compared with 41.0 for JiT-T2I-XXL/8++, while using 275 times less pretraining compute according to the paper.
• Controlled experiments show that text-only pretraining did not improve text-to-image generation.

The comparisons should be read within the paper’s evaluation setup. gFD is the reported generation-quality metric, and the numbers here describe the relative results provided by the authors rather than a universal ranking of all image-generation systems.

The model also explores image captioning. Instead of simply increasing the training budget, the researchers replace caption dropout with image-to-text supervision. This allows the same training setup to learn captioning while, according to the paper, retaining competitive text-to-image quality under the same pretraining budget.

Why it matters

Halcyon’s main contribution is a challenge to the assumed resource pipeline for text-to-image systems. If the application is centered on image-text generation, directly training on paired multimodal data may be a more targeted alternative to building or borrowing a large general-purpose language model. That could make the training objective easier to align with the final task.

The result does not show that language pretraining is useless in every setting. Halcyon is evaluated with a particular architecture, dataset, and compute comparison, and GPIC itself contains a very large number of paired examples. Data quality, caption diversity, instruction following, and performance on editing or compositional prompts still require broader validation.

The next question is whether early fusion can maintain its efficiency as models expand to more tasks. The paper’s joint image-to-text and text-to-image experiment also points toward multimodal systems that learn both directions without relying on a separate language foundation model.

Source: Hugging Face Daily Papers