Can One Algorithm Train a Foundation Model End to End?

Introduction

Foundation models are commonly built through a sequence of distinct stages. A model may begin with large-scale pretraining, proceed through midtraining, receive supervised fine-tuning, and finally undergo reinforcement learning. Each stage often brings its own data pipeline, objective, trainer, and engineering assumptions. The paper Training Foundation Models with a Single Algorithm questions whether these boundaries are truly necessary.

Its proposal is not to erase the differences between training data or learning goals. Instead, it looks for a common optimization language that can describe them. The authors show that suitable reinforcement-learning objectives can be derived as maximum-likelihood fitting to target distributions generated by the model. Under this formulation, RL can be handled in a form that is closely related to ordinary likelihood-based training.

One trainer, different sources of supervision

This perspective places pretraining, SFT, and RL inside a single likelihood-based algorithm. The stages remain different in terms of where their examples come from and what those examples are intended to teach, but they no longer require an obligatory change of optimization machinery.

The proposed unified system mixes static datasets with samples produced through RL within one trainer. Conventional staged training is still available: it becomes a particular choice of data schedule in the larger unified framework. The important change is conceptual. Instead of asking when to switch from one algorithm to another, researchers can ask which data should be presented, in what proportions, and at what point in training.

The training recipe becomes a data-allocation problem

The framework highlights several design choices:

• which static and generated datasets should be included;
• how much weight each source should receive;
• when generated samples should enter the training mix;
• how to balance specialized task gains with broader capabilities.

This does not mean that a single universal recipe will work for every model. It means that these choices can be expressed within a shared training process and compared more systematically. A staged pipeline is therefore treated as one point in a larger space of possible data-mixture policies.

Reported results and broader implications

Across controlled and real-world experiments, the authors report that unified training improves the accuracy-compute tradeoff and increases task coverage while preserving broader capabilities. The approach also provides a stronger starting point for continual learning. In practical terms, future training pipelines might be able to introduce new capabilities without treating supervised adaptation and reinforcement learning as entirely separate operations.

The broader contribution is a connection between mathematical objectives, training infrastructure, and capability development. It suggests that progress may come not only from inventing new loss functions, but also from understanding how different data distributions interact over the lifecycle of a foundation model.

The framework does not by itself solve data-quality problems, reward or target design, optimization stability, or forgetting. Those issues remain dependent on the selected data and scheduling policy. Its value is to offer a common foundation for studying them, and to make training design more flexible than a fixed sequence of algorithmic stages.

Source: Hugging Face Daily Papers