DiffusionART Makes Reinforcement Learning for Diffusion Models More Efficient

Introduction

Diffusion models can already produce high-quality images and videos, but improving properties such as visual quality, motion quality, or prompt alignment often requires more than likelihood training. Online reinforcement learning offers a direct way to optimize these properties: the model generates samples, a black-box reward evaluates them, and the policy is updated accordingly. The challenge is that a diffusion sample is produced through many denoising steps, making it difficult to assign a final reward to the decisions that created it.

A new paper from KlingAI Research presents DiffusionART, a reinforcement learning method designed to make this process more efficient, stable, and scalable. The approach is evaluated across image and video generation settings.

Key ideas

• A different view of credit assignment. Reverse-process methods such as Flow-GRPO must distribute a sample-level reward across multiple denoising steps. According to the paper, this can lead to inefficient training and weak credit assignment. DiffusionART organizes the reward signal around forward-process states instead.
• Aggregate before tilting. Rather than applying separate updates to positive and negative samples, DiffusionART first aggregates rewards from samples associated with the same prompt. This produces a shared direction that is intended to improve the prompt-level reward. The authors argue that this design removes, by construction, a hidden forward–reverse consistency force found in NFT-style objectives.
• Restoration without EMA tuning. DiffusionNFT can be stable when its exponential moving average, or EMA, is carefully tuned for off-policy training. DiffusionART adds a second update masked by a trust region, allowing restoration to adapt to the amount of model drift instead of relying on task-specific EMA settings.
• Flexible reward interfaces. The method does not require a differentiable reward model. It remains effective with fewer samples per prompt and includes DiffusionART-Pairwise for pairwise reward models.

Results and implications

The paper reports that DiffusionART trains efficiently and stably across image and video generation tasks without per-task tuning. On HunyuanVideo-13B, with visual and motion quality rewards weighted equally, it reaches the late-stage mean reward level of DiffusionNFT using 12.02 times fewer GPU hours. It reaches the corresponding level of Flow-GRPO-Fast with 11.72 times fewer GPU hours. The reported mean training rewards are also 28.8% and 53.0% higher than those baselines, respectively.

The importance of these results goes beyond a single benchmark. Video diffusion reinforcement learning is expensive because every training sample involves multi-step generation, and reward optimization may require repeated sampling. A method that extracts a more useful update from each prompt and batch can make compute a more manageable engineering constraint. DiffusionART therefore focuses on the structure of the reward update and on recovering from policy drift, rather than simply increasing sampling budgets.

The available material provides the method’s central motivation and headline results, while detailed task configurations, reward definitions, and implementation choices should be checked in the full paper and code. Still, the work offers a useful design lesson: stability in diffusion reinforcement learning may come not only from external averaging or stronger constraints, but also from how reward information is aggregated before the model is updated.

Source: Hugging Face Daily Papers