Skip to main content

​
Evolution Strategies (ES)

A class of black-box optimization algorithms inspired by natural evolution. At each generation, a population of parameter vectors is perturbed, evaluated by a fitness function, and the best performers recombine to produce the next generation. No gradients required.

​
Core algorithm (Natural ES variant)

Given policy parameters θ and objective F(θ):
  1. Sample n noise vectors ε₁…εₙ ~ N(0,I)
  2. Evaluate fitness Fᵢ = F(θ + σεᵢ) for each perturbation
  3. Update: θ ← θ + α · (1/nσ) Σ Fᵢεᵢ
The gradient estimator is equivalent to REINFORCE on the Gaussian-blurred objective — which is differentiable even when F is not.

​
The shared random seed trick (OpenAI ES, 2017)

The key scalability insight: if all workers synchronize their random seeds before optimization, each worker can reconstruct all other workers’ perturbations locally. Workers only need to communicate scalar fitness values — not parameter vectors or gradients. This enables linear scaling with number of workers. Communication cost: O(n) scalars per iteration regardless of model size.

​
Why ES vs RL

​
Key limitations ES has to overcome

Scalability: ES historically required population sizes ≥ 10,000. For modern LLMs (billions of parameters), full-rank perturbation is memory and compute prohibitive. Solutions developed across the three papers in this cluster:
  • OpenAI ES (2017): shared random seeds → scalar communication → 1,440-worker linear scaling
  • ES-LLM (2025): layer-by-layer in-place perturbation → population size 30 at 1B+ params → inference-only memory footprint
  • EGGROLL (2025): low-rank perturbations (rank-r matrices) → 100× GPU throughput improvement → 91% of batch inference throughput

​
Progression: from gaming to LLM fine-tuning

​
Key Results by Paper

​
OpenAI ES (Salimans et al., 2017) — 1703.03864

Qualitative finding: ES found diverse gaits (walking sideways, backwards) that TRPO never found — parameter-space exploration vs action-space exploration produces qualitatively different solutions. Additional techniques: antithetic sampling (evaluate ε, -ε pairs), fitness shaping (rank-transform returns), weight decay, virtual batch normalization.

​
ES-LLM (Qiu et al., 2025) — 2509.24372

Population size N=30, full-parameter LLM fine-tuning at 1B+ params. Fixed hyperparameters across all LLM families. Countdown task (symbolic reasoning, outcome-only rewards): RL required per-model grid search and still underperformed. ES used fixed hyperparameters across all. Reward hacking finding: RL optimizing a conciseness reward degenerated into brevity-hacking (near-empty responses). ES produced genuinely shorter responses without exploiting the reward — population-average optimization is structurally harder to hack than single-policy optimization. Memory advantage: no gradients → no optimizer states → inference-only memory footprint.

​
Relation to other concepts

  • Alternative post-training paradigm to RL methods like PPO, GRPO used in RLHF
  • Backpropagation-free: enables training non-differentiable architectures (integer quantized models)
  • Structural overlap with Context Engineering — both aim to reduce gradient/backprop dependency in parameter update