Stop Sampling Evenly: How NGU Helps LLMs Tackle Hard Problems

Introduction

Reinforcement learning is increasingly used to improve the reasoning abilities of large language models. Yet its benefits are not distributed evenly across a training set. In Learning to Solve Hard Problems in RL for LLMs by Never Giving Up, the authors report that RL often produces its clearest gains on easy problems that the model already handles relatively well, while progress on genuinely difficult problems remains limited.

They call this pattern the Matthew Effect in RL for LLMs, borrowing the phrase “the rich get richer” from economics and network science. Problems that are already easy to solve generate successful feedback more often, making them easier to reinforce. Hard problems, by contrast, may rarely produce a correct sample and therefore receive weaker learning signals.

Key ideas

• The issue is not only that hard problems need more compute. The paper argues that common RL procedures can worsen the imbalance. If every problem receives roughly the same number of generations, easy problems may continue consuming samples even after the model has effectively mastered them.
• NGU makes sampling adaptive. The rule behind Never Give Up is simple: continue generating samples for a problem until at least one correct answer is found. Once a problem succeeds, it no longer needs the same level of exploratory effort.
• Asynchronous RL enables reallocation. Problems progress independently. Easy examples can finish early, while difficult examples remain active and receive additional attempts. Compute is therefore allocated according to observed difficulty rather than a fixed quota.
• Off-policy behavior still matters. The study examines design choices related to asynchronous and off-policy training and develops best practices. NGU is not merely an instruction to generate indefinitely; its value depends on how those samples are incorporated into learning.

Results and implications

On the Deepscaler mathematics benchmark, NGU improves performance per unit of compute, with particularly strong benefits on harder problems. The coding task Manufactoria exposes another weakness of standard training. With GRPO and a per-test reward, a model may improve its average score by passing easy tests while failing to completely solve problems containing a mixture of easy and difficult tests. NGU instead keeps targeting the tests that remain unsolved, enabling iterative progress toward full solutions.

The broader contribution is a change in perspective. RL post-training is not only a question of designing better objectives or collecting more samples; it is also a scheduling problem. Which examples are allowed to keep consuming compute can shape the model’s final capability. A fixed batch and a fixed sample budget may be convenient, but they can spend too much effort confirming what the model already knows while starving the long tail of difficult cases.

The reported evidence covers mathematics and coding, so the approach still needs to be tested under other reward schemes, task distributions, and training scales. Even so, NGU offers a practical research direction: when a model encounters a hard problem, the training system should not give up merely because an average sampling budget has been exhausted. It should keep trying until the evidence says the problem has been solved.

Source: Hugging Face Daily Papers