Reinforcement-learning (RL) post-training can improve image quality, text rendering, and the ability to place several requested objects in a scene. Because these behaviors depend on the prompt, it is natural to assume that a model must learn complex transformations that adapt to each input. A new study offers a simpler account: much of the improvement may come from steering capabilities the pretrained model already has.
A small intervention in a frozen model
Rather than updating the full model, the researchers added a fixed set of steering vectors to the block outputs of a frozen diffusion transformer. The same vectors are used for every prompt, text token, and sampling seed, so they cannot make prompt-specific adjustments. The team fitted the vectors to match velocity predictions from public RL-tuned checkpoints, without using a reward signal in that fitting objective.
On SD3.5-M, the vectors contain 0.29 million parameters, compared with 18.8 million in the RL-tuned LoRA—a reduction of about 65 times. Despite this much smaller parameter count, the method recovered 90.0%–96.3% of the reward gains across six RL-tuned image models. On two video models, it recovered 89.8% and 111.9% of the gains, respectively. The latter result means the method exceeded the comparison checkpoint on the reported evaluation; it does not establish that it is better across every aspect of video generation.
The researchers also initialized the vectors at zero and trained them with online RL. On SD3.5-M, this approach achieved greater reward gains than weight-based RL on OCR, used to assess text rendering, and PickScore, a human-preference metric.
Why the result matters—and what it does not show
The findings question whether prompt-dependent improvements always require new, input-dependent computations. A constant intervention can substantially shift outputs, suggesting that the base model may already contain much of the necessary capability, while post-training changes how readily that capability is expressed. This offers a compact way to study model behavior and raises the possibility that some post-training methods could be made more parameter-efficient.
The evidence should not be read as proof that all RL gains are fixed shifts. The reported results concern particular models and reward metrics, and other tasks or prompt distributions may require input-dependent adaptation. Nor do reward improvements alone establish that every quality dimension or the diversity of generations is preserved. The cautious takeaway is that, in these experiments, fixed activation steering accounts for most of the measured gains—and that further evaluation is needed to map where this explanation holds.
Source: Hugging Face Daily Papers
Comments
Checking sign-in status...
Loading comments...