Jointly Aligning and Distilling Video Generators with DM-Align

Introduction

A practical video generator needs to satisfy several competing requirements: it should produce visually faithful clips, follow user intent, reflect human preferences, and do so with a manageable sampling cost. Existing training recipes often split these goals into separate stages. A model may first be distilled to reduce its sampling burden and then optimized with reinforcement learning, or it may be aligned first and compressed afterward. The first ordering can make reinforcement learning prohibitively expensive, while the second may cause the distilled model to lose its alignment capability or become unstable.

The arXiv paper “Joint Alignment and Distillation for Video Generation via Sample-Guided Distribution Matching” proposes a different organization of the training process. Its central idea is to perform distillation and preference alignment together within a unified distribution-matching framework, rather than treating them as disconnected procedures.

Key ideas

• A distillation direction is retained. Standard distribution matching narrows the gap between a reference or real model and the generated model. In this setting, the direction is intended to preserve clarity and fidelity while transferring behavior to a more efficient generator.
• DM-Align adds a preference direction. The method uses differences between sample distributions to construct a complementary gradient. This gradient pushes the generator toward outputs that are preferred by humans, rather than merely imitating the reference model.
• Preference pairs and group exploration are supported. For paired preference data, the approach draws inspiration from DPO. For multiple samples explored within a group, it takes cues from GRPO and uses their relative distributional information to guide optimization.
• The RL pipeline is simplified. According to the paper, the method avoids multi-step reward evaluation and the ODE-SDE conversions associated with conventional reinforcement-learning formulations.

Why it matters

The contribution is broader than adding another preference loss. It defines a shared update mechanism in which distillation protects generation quality and efficiency, while the alignment component deliberately reshapes the output distribution. Combining these directions in one stage could reduce the conflict between compression and preference optimization.

This design is potentially relevant to applications that need low-latency video generation without giving up visual quality or user-oriented behavior. It also offers a way to use preference signals more directly at the sample level, instead of relying on a long chain of reward modeling, trajectory evaluation, and policy updates.

The available material states that experiments on multiple foundation video models outperform standalone alternatives and sequential two-stage pipelines in both distillation quality and preference alignment. However, the abstract does not provide the specific datasets, metrics, training costs, or ablation details. Those details are important for judging how robust the gains are across different model families and preference sources.

Overall, DM-Align represents a shift from sequential training to joint optimization. It does not eliminate the need for careful preference data or evaluation, but it presents a promising route for making video generators faster and more aligned without forcing these objectives into separate, potentially conflicting stages.

Source: arXiv