Evolution Strategies Fine-Tuning: What Dropping Backprop Buys, and Where the Bill Moves

Evolution strategies fine-tuning is the oldest idea in the post-training conversation that has suddenly started producing current numbers. On a 15-turn Sudoku agent task, a Qwen3.5-4B tuned with evolution strategies (ES) fit in 8.41 GB of GPU memory where the matched GRPO baseline needed 58.88 GB, spent 9.4 EFLOPs against GRPO's 10.9, finished in 9.4 hours against 19.0 on a 4x H100 node, and scored 12.5 points higher. There is no backward pass anywhere in that loop. The model is only ever run forward.

The obvious skeptical reading of a backprop-free method is that the memory you save comes back as samples, and that the papers report the gigabytes while hiding the rollouts. It is the argument this piece set out to make. Checked against the papers, it is half wrong: on Sudoku and on the reasoning benchmarks the authors did price the compute, at matched FLOPs, and ES held up. The honest version of the skepticism is narrower and more useful. The cost that is still unpriced is environment time on the expensive task, the 27B web-agent result that carries the headline, and the result that is still unsettled is forgetting, where four 2026 papers reach different conclusions. This piece walks the mechanism, the evidence, the two open questions, and a decision procedure against the baseline this blog always starts from: LoRA first, then a policy gradient.

What evolution strategies actually do

RL post-training estimates a gradient of the reward with respect to the policy and steps along it. GRPO samples a group of completions per prompt, scores them, uses the group's reward spread as the baseline, and backpropagates a token-level objective. That backward pass is why the training stack holds activations, optimizer state, and usually a frozen reference model.

ES throws the gradient out. Take the current weights. Make a population of N copies, each with small Gaussian noise added to every parameter. Run each copy on the task and record one scalar reward per copy. Move the base weights toward the noise directions that scored well and away from the ones that scored badly. Repeat.

# theta: current weights; sigma: noise scale; alpha: step size; N: population
for step in range(T):
    seeds, rewards = [], []
    for i in range(N // 2):                     # mirrored (antithetic) pairs
        s = fresh_seed()
        eps = sigma * randn_like(theta, seed=s)
        rewards += [evaluate(theta + eps), evaluate(theta - eps)]
        seeds   += [(s, +1), (s, -1)]
    adv = rank_normalize(rewards)               # ranks, not raw rewards
    theta += alpha / (N * sigma) * sum(a * sign * randn_like(theta, seed=s)
                                       for a, (s, sign) in zip(adv, seeds))

Three details carry the scale results. Mirrored sampling evaluates each perturbation in both directions, which cuts the variance of the estimate for free. Rank normalization means one outlier reward cannot dominate a step. Seed replay means a worker never ships a perturbed copy of the weights: it ships a random seed and a scalar, and every worker regenerates the same noise locally. That last trick is why peak memory is one model's worth of inference and why the method parallelizes so cleanly; it is the property OpenAI's 2017 ES paper (Salimans et al., "Evolution Strategies as a Scalable Alternative to Reinforcement Learning") used to scale to over a thousand workers on simulated control.

The axis that matters is where credit is assigned. RL assigns it per token through a gradient. ES assigns it per episode through ranking. Every advantage and every cost below follows from that one difference.

The scale result that reopened the question

For a decade the reason nobody trained large models with ES was population size. Classic results used populations of 10,000 or more on networks with millions of parameters, and the variance of a perturbation estimate grows with dimension. A billion-parameter model looked hopeless.

"Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learning" (Qiu et al., Cognizant AI Lab; arXiv 2509.24372, ICML 2026) showed that assumption was wrong. With a population of 30, full-parameter ES on Qwen2.5 (0.5B to 7B) and Llama-3 (1B to 8B), the baseline families of its time, beat its RL baselines on the Countdown arithmetic task across models: Qwen2.5-7B reached 66.8% with ES against 57.5% with Dr.GRPO. The September 2025 preprint (v1) also claimed, in a section titled "ES is more sample efficient than RL," that ES needed "less than 20% of the training sample evaluations in most cases" to match RL. The current version (v3, July 2026) drops both the figure and sample efficiency from its list of advantages, which now reads tolerance to long-horizon rewards, robustness across base models, reduced reward hacking, and training stability. When the authors withdraw a number themselves, do not quote it.

The reference code became a maintained library, es-at-scale, in June 2026: Ray for distribution, vLLM for rollouts, no optimizer state, and a README stating the same implementation has trained models from 0.5B to 72B. The rule it states is the honest summary of the memory story: if the model fits in your GPUs under vLLM, it can be fine-tuned this way.

2026: agents, reasoning, and the bill that was actually paid

Two August 2026 preprints moved ES from arithmetic toys toward the workloads people care about. They share three authors (Zhi Zheng, Yunpeng Ba, Zhenkun Wang), so read them as one research program arguing from two angles, not as independent corroboration.

Agentic ESOpt (Zheng, Chen, Ba, Wang, Teh, Lee; arXiv 2608.17310) applies ES to long-horizon agents, with a cosine-decayed perturbation scale. Its claims: full-parameter optimization at inference-level memory, a black-box reward interface that composes with prompt-space methods such as skill evolution, and trajectory-level credit that does not decompose across turns. Its results, from the paper and the project repo:

• Sudoku, 15-turn setting, Qwen3.5-4B: 53.13% against 40.63% for agentic GRPO, a 12.5-point gain, with the ES advantage growing as the minimum solving horizon grows.
• Math and DocVQA, Qwen3.5-4B: 13.7 points over the base model on average and 8.3 over agentic GRPO.
• WebArena-Lite, Qwen3.5-27B: 29.47% to 36.16% with no skills, and 33.94% to 36.36% combined with Trace2Skill, at a population of 8.
• Heuristic design: better than its matched baseline in 28 of 36 test-time settings.

Understanding Evolution Strategies for LLM Reasoning (Ba et al.; arXiv 2608.27351) argues the reasoning side. It FLOP-matches ES (32 perturbation directions) against GRPO (8 responses per prompt). ES gets comparable or slightly lower Pass@1, but higher average Pass@16 and Pass@32, while GRPO's Pass@K falls below the base model in 15 of 18 comparisons. The mechanism is the test-time compute story run backwards: GRPO sharpens one policy and collapses entropy, so extra samples at inference become redundant; a population search keeps more of the model's reachable solutions alive. The paper proposes running GRPO first for Pass@1 and ES second for coverage. The coverage result is the one claim here with a second, separate group behind it: "Beyond the Best Guess" (Hayes, Meyerson, Schweighofer, Dailey, Hodjat, Miikkulainen, Qiu; arXiv 2608.12679, August 2026), from the Cognizant team behind the scale result, also reports consistently higher pass@k than RL and a broader output distribution.

Now the part the skeptical reading got wrong. Both papers report a matched-compute comparison. On Sudoku, Agentic ESOpt reports peak memory (8.41 GB against 58.88 GB), model FLOPs (9.4 against 10.9 EFLOPs), and wall-clock (9.4 against 19.0 hours on 4x H100 80GB). The reasoning paper matches FLOPs by construction. The intuition that "no backprop means many more forward passes" ignores that a backward pass costs roughly twice a forward pass and that GRPO also runs a reference model. Dropping both pays for a population of 32 almost exactly.

What is still unpriced: the slow environment

Look at what "model FLOPs" counts. It counts the GPU work of running the model. It does not count the time the environment takes to answer. On Sudoku the environment is a few lines of Python and returns instantly, so model FLOPs is the whole bill. On WebArena-Lite a rollout is a browser session against a live web application, and the environment's latency, statefulness, and reset cost dominate.

That is the setting that carries the paper's headline, the 27B result, and it is the one setting with no rollout count and no wall-clock in the paper. The population is only 8 there, which is itself a signal: on a slow environment you cannot afford 32 copies per step. So the claim the evidence supports is precise. On a cheap, fast, parallel environment, ES matches GRPO's compute and beats its memory and wall-clock. On a slow, stateful environment, nobody has published the ledger.

Parallelism is ES's strongest answer here, and it is real: each population member is independent and reports back one scalar, so wall-clock can be bought with width. But that assumes the environment parallelizes. A rate-limited external API, a shared staging database, or a production system with side effects does not run 32 copies the way a simulator does. Whether your rollouts parallelize is a property of your environment that no benchmark paper can tell you.

One more honest-baseline note on the memory figure. The paper does not say whether its 58.88 GB GRPO baseline was full-parameter or adapter-based, but the released Sudoku training script answers it: the GRPO baseline loads the full BF16 model, hands every parameter to AdamW, enables gradient checkpointing, and keeps a second frozen copy as the reference model. No adapter anywhere. So the comparison is full-parameter ES against full-parameter GRPO, which is fair as far as it goes, and it is not the comparison most teams would run. A LoRA adapter keeps optimizer state for well under 1% of the weights, and on a 4B model that closes much of the gap. Nobody has published ES against LoRA-GRPO on the same task. ES's memory advantage is decisive when full-parameter updates are the point, or when the model is so large that even an adapter's backward pass does not fit. Against the baseline most teams actually run, it is unmeasured.

Reward hacking cuts both ways

The Qiu paper's most interesting result is not a score. On a conciseness reward, GRPO with a KL coefficient of 0.0 or 0.01 collapsed into short strings of nonsensical symbols that the proxy loved; ES reached comparable reward without degenerating, with no KL penalty at all. The mechanism is plausible: a gradient points straight at the nearest direction that raises the proxy, which makes it an excellent instrument for finding a proxy's sharp artifacts, while a ranked population of whole-model perturbations cannot zero in on a razor-thin exploit. This is the rare case that refines this blog's standing position that the reward signal, not the optimizer, sets the ceiling: here the optimizer changes the hacking surface.

It does not remove it. ES optimizes a black-box scalar with no notion of what the score was supposed to mean. A sharp artifact is hard for ES to find; a broad one is easy. A timeout that counts as success, a success check that only inspects the final page title, an environment that can be driven into a rewarded state without doing the task: these are wide basins, and a population search will walk into them as readily as into a real improvement. The RLVR lesson applies in full: the reward-hacking surface moves into your verifier code, and on an agent task the verifier is your entire harness. Audit it before you hand it to an optimizer that will search it exhaustively.

Forgetting: four papers, three answers

This is where the literature is genuinely unsettled, and where a practitioner most needs to measure rather than read.

• "Evolutionary Strategies lead to Catastrophic Forgetting in LLMs" (Abdi et al.; arXiv 2601.20861, January 2026) finds ES reaches near-GRPO scores on math and reasoning at comparable compute but with significant forgetting of prior abilities, and traces it to updates that are much less sparse and have orders of magnitude larger L2 norm than GRPO's.
• "Matching Accuracy, Different Geometry" (Hoy, Wang, Pan; arXiv 2604.01499, COLM 2026) finds ES matches or beats GRPO on single-task accuracy, but with much larger parameter changes and broader off-task KL drift. Oddly, the two solutions are linearly connected with no loss barrier, though their update directions are nearly orthogonal.
• "Overcoming Forgetting in LLM Fine-Tuning with Evolution Strategies" (Schweighofer, Hayes, Dailey, Miikkulainen, Qiu; arXiv 2605.30148, May 2026), from the group behind the scale result, reframes it as performance drift rather than irreversible forgetting, shows GRPO drifts too, and traces ES drift to a random walk in weight directions the reward does not constrain. On Countdown at population 30, average prior-task accuracy (HellaSwag, PIQA, ARC-Challenge, MMLU-Pro) fell 5.2 points. Raising the population to 128 cut that to 0.6 at 4 to 8 times the compute; their Anchored Weight Decay, which pulls the weights back toward the initial model, cut it to 0.9 for 1 to 2% extra runtime.
• Ba et al. (the reasoning-coverage paper) report no catastrophic forgetting on held-out GPQA, MBPP, CSQA, and Countdown, argue that the gains come from a sparse subset of larger updates ("functional sparsity"), and contend that Abdi et al.'s evidence came from a single model, task, and held-out benchmark and is hard to separate from overfitting.

The reconciliation that fits all four: vanilla ES has no leash. GRPO carries a KL penalty to a reference model; plain ES carries nothing, so weight directions the reward does not care about wander. Whether that wandering shows up as forgetting depends on population size, step count, and whether you add an anchor. Forgetting is a property of the configuration, not of the method. That is better news than "ES forgets," and worse news than "ES is fine," because it means the only way to know is to measure your run.

The tooling that attacks the cost

Three lines of work exist because the cost is real, and each targets a different bottleneck:

• EGGROLL (Sarkar, Fellows, Duque, Letcher, and co-authors; arXiv 2511.16652) represents each perturbation as a rank-r matrix instead of dense noise, reporting a hundredfold training speedup for billion-parameter models at large populations and up to 91% of pure batch-inference throughput. Reach for it when you need a population far above 30.
• Quantized Evolution Strategies (Xu, Schweighofer, Miikkulainen, Qiu; arXiv 2602.03120) fine-tunes directly in a quantized weight space with error feedback and seed replay. This matters if you are fine-tuning a 4-bit-native model with no higher-precision checkpoint to merge into: ES does not need a differentiable, continuous base.
• ESSAM (arXiv 2602.01003) adds sharpness-aware updates to push ES toward flatter minima, a third acknowledgment that plain ES has a generalization problem worth engineering against.

When ES is the right fine-tuner

The standing line holds: vanilla LoRA first, data beats method, and do not fine-tune to inject knowledge. For preference data, DPO and its descendants remain the default; for checkable reasoning tasks, GRPO with a verifier remains the default. ES earns its place in three cases: the reward is non-differentiable or only available at the end of an episode, full-parameter updates are required on hardware that cannot hold a backward pass, or Pass@K coverage is the objective. It is not yet a production post-training recipe anywhere public; the evidence stops at 27B on research benchmarks.

Run this before committing a budget:

1. Baseline LoRA plus GRPO on your task first. Record the score and the total environment interactions it took. If it clears your bar, stop.
2. Classify the environment, not just the reward. Measure one rollout's wall-clock and whether 30 copies can run concurrently. A fast, parallel environment is where ES's compute parity has been shown. A slow or stateful one is unpriced territory; budget a small pilot before believing any projection.
3. Budget in environment time and FLOPs, never in gigabytes. Set a target score from step 1, run a small-population ES pilot, and extrapolate total rollouts and wall-clock to that target. Memory is a threshold: if the backward pass already fits, ES's memory saving buys you nothing you will feel.
4. Pin a public retention benchmark before you start. Score the base model on MMLU-Pro and ARC-Challenge (the held-out sets the drift study used) and re-score after tuning. A target-task gain with a retention loss is a decision, not a win. Turn on an anchor (Anchored Weight Decay or an explicit pull toward the initial weights) by default.
5. Audit the reward as attack surface. List every way the scalar can rise without the task being done: timeouts, loose success checks, degenerate environment states. Close them, and keep a private held-out eval the search never touches.
6. Report Pass@K alongside Pass@1. If the task has many valid solutions, a Pass@1-only comparison understates ES. Consider GRPO first and ES second before treating the choice as either-or.
7. Pick the variant for your bottleneck. Population too small for the task: EGGROLL. Base only ships quantized: QES. Generalization gap: a sharpness-aware variant.

The honest verdict

Evolution strategies fine-tuning is no longer a curiosity. Full-parameter, backprop-free updates work at billion-parameter scale with tiny populations, they fit in inference memory, and on a cheap environment they match GRPO's compute while beating its memory and wall-clock. The method also resists the sharp reward hacks that gradients find. Those are measured results, not pitches.

The skeptic's job is to put the asterisks in the right places, and they are not where the obvious objection puts them. The first asterisk is the slow environment: the 27B web-agent headline comes without its rollout bill, and that bill is the one that decides real agent training. The second is forgetting, which four papers disagree on because vanilla ES has no leash, and which an anchor largely fixes. Run the matched comparison on your own environment, measure retention on a public benchmark, and audit the reward. Do that and ES is a real addition to the fine-tuning toolkit. Skip it and "inference-level memory" is one more half-reported number.

Key Takeaways

1. ES updates every weight with no backward pass. It ranks a population of perturbed copies by a scalar reward and steps toward the winners, so memory drops to inference level and the reward never has to be differentiable.
2. Small populations work at scale. Qiu et al. (ICML 2026) fine-tuned 0.5B to 8B models with a population of 30 and beat RL on Countdown (66.8% against 57.5% for Qwen2.5-7B); the maintained es-at-scale library reports runs up to 72B.
3. On a cheap environment the compute is priced, and ES holds up. On 15-turn Sudoku, Agentic ESOpt used 8.41 GB against 58.88 GB, 9.4 against 10.9 EFLOPs, and 9.4 against 19.0 hours, and scored 12.5 points higher. Skipping the backward pass and the reference model pays for the population.
4. The unpriced cost is environment time. The 27B WebArena-Lite result (29.47% to 36.16%) reports no rollout count or wall-clock, and slow, stateful environments are where rollouts are expensive and parallelism breaks down.
5. ES reaches broader coverage than GRPO. At matched FLOPs it gives higher Pass@16 and Pass@32 while GRPO's entropy collapse pushes Pass@K below the base model in 15 of 18 comparisons; GRPO first, ES second is a credible hybrid.
6. Gradient-free changes the reward-hacking surface but does not remove it. ES resisted a sharp conciseness hack that broke GRPO, yet a black-box scalar with a broad exploit (a timeout counted as success) will still be found.
7. Forgetting depends on configuration. Four 2026 papers disagree; vanilla ES has no leash, and Anchored Weight Decay cut average prior-task drift from 5.2 points to 0.9 for 1 to 2% extra runtime. Measure retention on a public benchmark before and after every run.