AV-GRPO Separates Reinforcement Learning for Joint Audio-Video Generation

Why this matters

Joint audio-video generation is moving beyond producing two streams at the same time. A useful system must make the visuals, sound, prompt, and timing agree. Reinforcement-learning post-training appears suitable for improving these properties, but it introduces a difficult credit-assignment problem: when several heterogeneous rewards are combined, the model may not know whether an update improved the video, the audio, the text alignment, or synchronization.

AV-GRPO proposes a modality-anchored online diffusion reinforcement-learning framework designed around that problem. Rather than optimizing both modality towers as if they followed identical dynamics, the method decomposes the training process and uses one modality as an anchor when collecting and comparing rollouts.

Main components

• Modality-anchored rollouts: An anchored modality helps stabilize the difficulty of comparisons and separates learning signals that would otherwise be mixed together. This is intended to make reward attribution more precise.
• Trajectory-locked frozen-tower optimization: When one modality is being optimized, the other tower is frozen and the relevant trajectory is locked. This reduces the cost of jointly updating two divergent modality systems and reassigns credit to the branch being trained.
• Adaptive objectives and perturbations: Audio and video do not respond identically to diffusion perturbations or optimization schedules. AV-GRPO therefore adjusts objectives and perturbation strengths according to modality-specific behavior.

The authors also introduce 5DAV, a training dataset that decouples samples across five dimensions and allows difficulty to be controlled systematically. The dataset is positioned as a way to study separate modality capabilities and cross-modal synchronization rather than treating every example as an undifferentiated preference pair.

Results and broader significance

The reported evaluation uses JavisBench and VABench, with LTX-2.3 serving as the comparison point under both LoRA and full-parameter fine-tuning. According to the supplied abstract, AV-GRPO improves generation quality, semantic alignment, and audio-video synchronization over the baseline. Ablation studies are reported to support the contribution of the individual design choices.

The broader contribution is conceptual as much as computational. The work treats multimodal reinforcement learning as a problem of signal separation and fair comparison, not merely as a matter of adding more reward terms. Anchoring one modality can make optimization more interpretable, while freezing the other tower can reduce unnecessary interference and cost. The approach is also described as practical for training the 22B LTX-2.3 model with LoRA or full-parameter updates on a limited multi-GPU setup.

The available material does not provide detailed metric values, reward-model specifications, or a complete account of generalization to longer clips and more complex soundscapes. Those questions require checking the full paper and released implementation. Still, AV-GRPO offers a structured route for improving joint generation without forcing audio and video to share every training decision.

Source: Hugging Face Daily Papers