Turns Matter: Turn-Level Credit for Streaming Video RL

Streaming video understanding is not the same as answering a question after watching an entire clip. A model receives visual information incrementally and must repeatedly decide whether to respond now or keep observing. This creates a sequence of interaction turns, making the assignment of reinforcement-learning credit especially important.

The problem with one advantage for a whole trajectory

Recent work has applied Group Relative Policy Optimization, or GRPO, to streaming video tasks. However, existing approaches often retain a trajectory-level advantage: the final feedback from an interaction is shared across all decisions made along the way. While straightforward, this can produce poorly aligned learning signals.

Suppose a model gives a successful answer early and then produces another unnecessary reply. A shared positive advantage may reinforce both behaviors, including the redundant one. Conversely, if an early response fails, a later follow-up that would have corrected or enriched the answer can be penalized by the same low trajectory-level signal. The challenge is therefore not only how to design rewards, but also how to associate them with the specific response-or-wait decision that produced them.

Two components of TurnWise

The paper introduces TurnWise, a turn-level GRPO framework designed around the structure of streaming interaction. It has two main components:

• Turn-level credit assignment: TurnWise scores a prefix after completing it with silence, meaning that the evaluated decision is treated as the point from which the model no longer actively responds. The remaining return from that decision is then used to construct a group-relative advantage. This separates the signal for a turn from rewards that occurred earlier in the trajectory.
• Turn-mean policy optimization: The method first averages token losses within each turn and then averages across turns. Long answers therefore do not dominate training simply because they contain more tokens, while response and waiting decisions receive more balanced representation.

Together, these mechanisms address two separate sources of mismatch: which turn should receive credit, and how turns with different output lengths should contribute to the update. TurnWise does not require a learned critic or additional rollouts.

Why it matters

The broader contribution is a change in the basic unit of optimization for streaming video reinforcement learning. In a system that must continuously decide when to speak and when to wait, timing is part of the task—not merely a side effect of answer generation.

The reported experiments show that TurnWise outperforms prior reinforcement-learning approaches for streaming video understanding. The result suggests that more carefully structured credit assignment and loss aggregation can improve streaming policies without simply adding more reward rules. The same perspective may be relevant to real-time video assistants, persistent perception systems, and other agents that alternate between waiting and acting.

Source: Hugging Face Daily Papers