Why Kernel Alignment Matters in Asynchronous RL

Introduction

Asynchronous reinforcement learning is attractive because rollout workers and the learner no longer have to take turns. Rollouts can continue while the learner performs updates, improving hardware utilization and shortening the time needed to collect experience. The trade-off is that a sample may be generated by an older policy than the one that eventually trains on it.

A new vLLM Blog post discusses REINFORCE Zero, a recipe developed by NVIDIA Research and Inferact. Its central argument is that asynchronous RL contains two different gaps, and treating them as one can lead to unstable training.

Two sources of mismatch

Suppose a rollout is generated with weight version v and consumed by the learner at version k. Token-level importance sampling can correct the policy difference by comparing the learner’s probability with the probability recorded by the rollout engine. This is the familiar off-policy correction.

However, the learner and rollout engine can produce different log probabilities even when they use the same model weights. Differences in kernels, numerical precision, batching, or execution paths can create this second gap. The post calls it kernel mismatch.

The distinction matters because the two effects behave differently. Weight lag generally changes smoothly as the policy moves through successive updates. Kernel mismatch can fluctuate with batch composition and numerical execution, creating large excursions that importance sampling alone does not reliably control. In the cited experiment, the peak-to-median ratio of an importance-weight deviation measure was 1.50 with aligned kernels, compared with 442.54 with standard kernels.

The REINFORCE Zero recipe

The method has three practical components:

• Align rollout and learner kernels so that the same weights and tokens produce the same log probabilities.
• Apply token-level importance sampling without clipping, truncation, or sequence-rejection gates.
• Optimize plain REINFORCE, using each response’s reward relative to its group mean as the advantage and requiring no value model.

Unclipped ratios preserve the expected policy-gradient correction when the probabilities are exact. Clipping can reduce variance, but it also introduces bias. REINFORCE Zero’s claim is not that unclipped importance sampling is universally stable; rather, it becomes practical after the much noisier kernel mismatch has been removed.

Reported results

With standard kernels, the rollout-to-learner KL increased substantially during training, while AIME24 pass@1 declined from 34.27% to 13.96%. The largest gradient norm reached 38.27. With aligned kernels, the KL remained flat and REINFORCE Zero rejected no sequences. A sequence gate could stabilize the standard-kernel run, but it rejected 24.48% of sequences near the end.

The method also continued learning at policy lag 64 with Qwen3-4B, Python tools, and a 16K context. Across 5,000 updates, mean reward rose from 0.287 in the first 100 updates to about 0.78. In a 16-GPU comparison over 12,000 updates, aligned asynchronous training achieved 2.40× the training-step throughput of aligned synchronous training. On a 1.5B reasoning task, asynchronous runs at lag 2 and lag 8 reached results close to synchronous training.

The broader implication is a systems lesson: off-policy correction cannot compensate for inconsistent probability computation indefinitely. For tool use, long-context agents, and mixture-of-experts systems, aligning kernels may be more valuable than adding increasingly aggressive filtering rules. The cost is engineering effort for each model and hardware stack, so the approach should be evaluated together with alignment overhead and task-specific quality.

Source: vLLM Blog