Why Evolution Strategies Cover More LLM Reasoning Paths Than GRPO

Introduction

Post-training for large language model reasoning is often evaluated through two related but distinct metrics. Pass@1 asks whether the first sampled answer is correct, while Pass@K asks whether at least one correct answer appears across multiple samples. Group Relative Policy Optimization, or GRPO, is effective at improving single-attempt performance, but concentrating training on highly rewarded behaviors can also narrow the set of reasoning paths the model continues to explore.

The paper Understanding Evolution Strategies for LLM Reasoning investigates Evolution Strategies (ES) alongside GRPO to determine whether ES is simply a memory-efficient alternative or a genuinely different optimization paradigm. Its findings support the latter interpretation.

Key findings

• ES broadens reasoning coverage. Both the theoretical analysis and experiments suggest that ES preserves a wider range of candidate solution paths. This broader coverage is especially relevant to repeated-sampling settings, where it can translate into stronger Pass@K performance.
• Verifier-aware diversity matters. The authors study Jensen-Shannon diversity after projecting the ES population through a verifier. Their analysis connects higher verifier-projected diversity with better Pass@K results. The relevant diversity is therefore not just variation in wording; it concerns distinct candidate solutions that retain task-level value under verification.
• GRPO can experience entropy collapse. The experiments show that GRPO tends to make the output distribution more concentrated during optimization. This may help the model settle quickly on a strong answer pattern, but it can reduce exploration of alternative solutions. ES, by contrast, improves Pass@1 while also reaching higher Pass@K than GRPO.
• The methods can be combined. A sequential GRPO-ES strategy is introduced to combine GRPO’s strength on first-attempt accuracy with ES’s ability to preserve or expand solution coverage.
• Large parameter movement can be functionally sparse. ES may cause substantial drift across the full parameter set, yet the task-performance gains are attributed mainly to a sparse subset of larger-magnitude updates. Broad parameter movement does not necessarily mean broad behavioral change. Held-out evaluations further indicate that the drift does not inevitably produce catastrophic forgetting.
• Population size should follow model scale. The study reports that larger language models may work better with smaller ES populations, suggesting that population hyperparameters should not be transferred mechanically from smaller models.

Why it matters

The work places ES on a different conceptual footing from GRPO. Its value is not limited to lower memory requirements: ES offers an optimization route that explicitly benefits from maintaining multiple promising reasoning directions. For mathematics, coding, and other verifiable tasks, first-answer accuracy is important, but the ability to discover a correct answer among several attempts can define the practical ceiling of a system.

The study also argues for broader evaluation. Looking only at Pass@1 can hide a contraction of the model’s reasoning boundary. A training recipe that improves the most likely answer while suppressing alternatives may appear successful while losing useful capabilities. Monitoring Pass@K, output entropy, and verifier-aware diversity together could provide a more complete picture. In practice, GRPO and ES may be best viewed as complementary tools whose roles depend on whether a task prioritizes immediate reliability, exploration, or both.

Source: Hugging Face Daily Papers