> ## Documentation Index > Fetch the complete documentation index at: https://vietbui.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # Evolution Strategies (ES) > A class of black-box optimization algorithms inspired by natural evolution. At each generation, a population of parameter vectors is perturbed, evaluated by… # Evolution Strategies (ES) A class of black-box optimization algorithms inspired by natural evolution. At each generation, a population of parameter vectors is perturbed, evaluated by a fitness function, and the best performers recombine to produce the next generation. No gradients required. ## Core algorithm (Natural ES variant) Given policy parameters θ and objective F(θ): 1. Sample n noise vectors ε₁...εₙ \~ N(0,I) 2. Evaluate fitness Fᵢ = F(θ + σεᵢ) for each perturbation 3. Update: θ ← θ + α · (1/nσ) Σ Fᵢεᵢ The gradient estimator is equivalent to REINFORCE on the Gaussian-blurred objective — which is differentiable even when F is not. ## The shared random seed trick (OpenAI ES, 2017) The key scalability insight: if all workers synchronize their random seeds before optimization, each worker can reconstruct all other workers' perturbations locally. Workers only need to communicate scalar fitness values — not parameter vectors or gradients. This enables **linear scaling** with number of workers. Communication cost: O(n) scalars per iteration regardless of model size. ## Why ES vs RL | Property | ES | RL (policy gradient) | | -------------------- | ------------------------------------------ | -------------------------------------------- | | Gradient computation | None (inference only) | Backpropagation required | | Reward structure | Response-level only | Token-level credit assignment needed | | Long-horizon rewards | Naturally tolerant | High variance, difficult credit assignment | | Reward hacking | Harder (optimizes population distribution) | Easier (optimizes single policy) | | Parallelism | Linear speedup with workers | Limited by sequential value function updates | | Exploration | Parameter space | Action space | | Run consistency | High | Low (sensitive to hyperparameters) | ## Key limitations ES has to overcome **Scalability**: ES historically required population sizes ≥ 10,000. For modern LLMs (billions of parameters), full-rank perturbation is memory and compute prohibitive. **Solutions developed across the three papers in this cluster:** * OpenAI ES (2017): shared random seeds → scalar communication → 1,440-worker linear scaling * ES-LLM (2025): layer-by-layer in-place perturbation → population size 30 at 1B+ params → inference-only memory footprint * EGGROLL (2025): low-rank perturbations (rank-r matrices) → 100× GPU throughput improvement → 91% of batch inference throughput ## Progression: from gaming to LLM fine-tuning | Paper | Year | Scale | Result | | --------- | ---- | -------------------- | ---------------------------------------------------------- | | OpenAI ES | 2017 | \~millions of params | Competitive with A3C on Atari; 3D humanoid in 10 min | | ES-LLM | 2025 | 1B+ params | Outperforms GRPO/PPO on Countdown across all LLM families | | EGGROLL | 2025 | Hyperscale | Pretrains integer RNNs; competitive with GRPO on reasoning | ## Key Results by Paper ### OpenAI ES (Salimans et al., 2017) — 1703.03864 | Task | Result | | ---------------------- | ------------------------------------ | | MuJoCo 3D Humanoid | Solved in 10 minutes (1,440 workers) | | Atari (1 hr ES) | Better than 1-day A3C on 23/51 games | | MuJoCo data efficiency | Matches TRPO with ≤10× more data | Qualitative finding: ES found diverse gaits (walking sideways, backwards) that TRPO never found — parameter-space exploration vs action-space exploration produces qualitatively different solutions. Additional techniques: antithetic sampling (evaluate ε, -ε pairs), fitness shaping (rank-transform returns), weight decay, virtual batch normalization. ### ES-LLM (Qiu et al., 2025) — 2509.24372 Population size N=30, full-parameter LLM fine-tuning at 1B+ params. Fixed hyperparameters across all LLM families. **Countdown task (symbolic reasoning, outcome-only rewards):** | Model | Original | Best RL | ES (ours) | | ------------- | -------- | ------- | --------- | | Qwen-2.5-0.5B | 0.1% | 13.5% | **14.4%** | | Qwen-2.5-1.5B | 0.7% | 31.0% | **37.3%** | | Qwen-2.5-3B | 10.0% | 43.8% | **60.5%** | RL required per-model grid search and still underperformed. ES used fixed hyperparameters across all. Reward hacking finding: RL optimizing a conciseness reward degenerated into brevity-hacking (near-empty responses). ES produced genuinely shorter responses without exploiting the reward — population-average optimization is structurally harder to hack than single-policy optimization. Memory advantage: no gradients → no optimizer states → inference-only memory footprint. ## Relation to other concepts * Alternative post-training paradigm to RL methods like PPO, GRPO used in RLHF * Backpropagation-free: enables training non-differentiable architectures (integer quantized models) * Structural overlap with [Context Engineering](/concepts/context-engineering) — both aim to reduce gradient/backprop dependency in parameter update