Training–Inference Mismatch in LLM RL: Origins and Remedies

Introduction

On-policy reinforcement learning for large language models assumes that responses are generated by the same policy that is being optimized. Production-style training systems rarely satisfy that assumption exactly. A high-throughput inference engine generates responses, while a separate trainer computes losses and gradients. The two components may differ in numerical behavior, and asynchronous execution may leave the sampler using older model weights.

The result is training data that is not truly on-policy. The paper from a George Washington University team asks a practical question: should every form of mismatch be corrected in the same way, or does the origin of the error determine the remedy?

Main findings

• Two distinct sources: By evaluating the trainer at the sampler’s weights, the authors split each sampled-token error into numerical discrepancy from the inference engine and policy lag between the sampler and trainer.
• Numerical discrepancy is comparatively stable: This component remains approximately constant during training. It therefore acts like a persistent bias in the update. Centering the score at each prefix removes its leading effect.
• Policy lag creates a feedback loop: Trainer updates move the current policy away from the sampler. The resulting off-policy bias enters the next update, which can enlarge the subsequent lag. The problem is therefore dynamic rather than a fixed offset.
• Importance weighting has a limit: Importance ratios can interrupt the loop while they remain concentrated. On Qwen3-1.7B, the correction fails when the ratio’s effective sample size falls below one half.

The DLC approach

The study derives discrepancy-lag correction, or DLC, from this decomposition. First, it centers the numerical discrepancy to reduce its persistent bias. It then clips the lag-related importance ratio using the sampler’s weights. Finally, if the effective sample size drops below one half, DLC resynchronizes the sampler instead of continuing to rely on increasingly unreliable ratios.

In experiments with Qwen3-0.6B and Qwen3-1.7B, DLC kept the KL divergence from stale-sampler training within twice the KL divergence observed between two independent on-policy runs. The resynchronization step also prevented collapses that token-level corrections alone failed to avoid. Notably, measuring KL only on the sampler’s prefixes did not necessarily expose those failures in time.

Why it matters

The paper’s broader contribution is diagnostic. Training–inference mismatch should not be treated as a single, undifferentiated source of noise. A relatively fixed numerical bias calls for centering, while policy lag evolves with the trainer’s own updates and requires ratio control plus synchronization.

For LLM reinforcement-learning infrastructure, this suggests monitoring more than rewards or a single KL statistic. The concentration of policy ratios and their effective sample size can provide an earlier warning that importance correction is becoming unreliable. The findings do not establish a universal guarantee for every RL algorithm or model, but they offer a useful way to connect the cause of mismatch with the appropriate stabilization mechanism.

Source: Hugging Face Daily Papers