🧬Fully Understand Evolution Strategies (ES) for LLM Reasoning
ES can post-train LLMs without backprop. Our study finds that ES:
🔍 Explores higher Pass@K than GRPO
💥 Still sparse functional updates, with no catastrophic forgetting.
⚙️ Requires fewer samples as LLMs get larger
We summarize the paper through 3 research questions 👇
1️⃣ Does ES exhibit the same post-training characteristics as GRPO?
ES maintains broader reasoning coverage!
Its population-based parameter-space exploration induces diverse reasoning behaviors: ES improves Pass@1 while achieving stronger large-K Pass@K, without the same entropy collapse observed in GRPO.
The two are also complementary. 🔀 Sequential compositions—ES→GRPO and GRPO→ES—combine GRPO’s strength in Pass@1 with ES’s gains in Pass@K, yielding better Pass@1–Pass@K trade-offs.
➡️ ES preserves broader coverage; combining it with GRPO can capture both strengths.
2️⃣ Does ES necessarily cause catastrophic forgetting?
No. Although ES induces substantial whole-model parameter drift, its task-relevant effects are concentrated in a small subset of larger-magnitude updates.
Most parameter changes contribute little after perturbation cancellation, revealing surprising functional sparsity. Held-out capabilities also remain largely preserved under appropriate training settings.
➡️ Large parameter movement does not necessarily imply widespread functional change—or catastrophic forgetting.
3️⃣ What hyperparameter settings and estimators make ES effective and scalable?
We find three practical takeaways:
✅ Z-score reward normalization is a key ingredient for effective ES training.
✅ Larger pretrained models require smaller population sizes for effective optimization.
✅ For discrete reasoning rewards, the two-point estimator commonly favored in zeroth-order SFT provides no advantage.
💡 Takeaway: ES is not merely a memory-efficient, gradient-free alternative to GRPO. Its broader reasoning coverage, functionally sparse updates, and distinct scaling behavior position it as a distinct paradigm for LLM reasoning post-training.
📄 arXiv:2608.27351