From RLHF to RLVR: How RL Post-Training Actually Evolved

Almost every step in the history of RL post-training has been mistaken for an algorithm advance when it was really a change in where the reward comes from. PPO to DPO looked like a new optimizer; it was mostly a way to delete the reward model. RLHF to RLVR looked like a new training loop; it was mostly swapping a learned, gameable reward for a verifier that is much harder to game. Track the optimizers and the field is a churn of acronyms: PPO, DPO, IPO, KTO, SimPO, GRPO, DAPO, GSPO, and a new one next month. Track the reward signal and it is a single coherent argument: the quality of post-training is set by the quality and integrity of the reward, and the algorithm is the plumbing that delivers it.

This is the map, drawn along that axis. It is the conceptual companion to the hands-on pieces in this cluster: the RLVR tutorial builds a verifiable-reward loop end to end, the reasoning-models explainer covers why thinking longer works at all, and the neural-symbiosis research piece places the whole family inside the bounded-feedback-loop frame this article assumes. The goal here is narrower: why each transition happened, what it fixed, and what it quietly left broken.

The starting point: RLHF and the learned reward

Post-training begins after pretraining and supervised fine-tuning have produced a model that can complete text but does not reliably do what a user wants. RLHF was the first scalable answer. Collect human preference comparisons (A is better than B), train a reward model to predict those preferences, then optimize the policy against that reward model with PPO, held near the reference model by a KL penalty so it does not drift into degenerate text.

It worked, and it had a structural weakness that everything downstream is a response to. The reward model is a learned approximation of human preference, and the policy is an optimizer pointed straight at it. Optimize hard enough and the policy finds the reward model's errors rather than the humans' intent: reward hacking, the canonical failure being length and sycophancy, where longer and more agreeable answers score higher whether or not they are better. PPO also carries real operational weight, since you are training four models at once (policy, frozen reference, reward model, and a value/critic network), which is memory-heavy, finicky, and expensive enough that most teams never ran it properly.

Two branches grow out of those two problems: one attacks the source of the preference labels, the other attacks the machinery. A third, later branch attacks the premise that the reward has to be learned at all.

Branch one: change where preferences come from

Human labeling is slow, expensive, and inconsistent. Constitutional AI (Bai et al., Anthropic, arXiv 2212.08073) replaced most of the human labels with AI-generated ones, an approach the literature generalized as RLAIF: a model critiques and revises its own outputs against a written set of principles (a "constitution"), and those AI preferences train the reward model. The signal still flows from human values, but indirectly, through the principles, which makes it cheaper to scale and easier to audit than a pile of individual labels. You can diff a constitution. You cannot diff a hundred thousand crowdworker judgments.

The branch is alive in 2026 and has moved from rules toward reasons. Anthropic's January 2026 constitution revision replaced a list of behaviors with an explicit four-tier priority ordering (broad safety, broad ethics, company policy, then helpfulness) plus the rationale behind each rule, giving the model a decision procedure rather than a lookup table. OpenAI's deliberative alignment is the same idea rebuilt for reasoning models: the safety specification is handed to the model to reason over at inference time instead of being compressed entirely into weights, which the reported results tie to better jailbreak robustness with fewer over-refusals.

Both are real improvements to how principles are expressed. Neither changes the load-bearing fact: the reward is still a learned proxy, and a constitution-trained reward model can be gamed the same way a human-trained one can. This branch made the signal cheaper and more auditable. It did not make it incorruptible.

Branch two: delete the reward model

The other branch asked whether the explicit reward model is necessary at all. Direct Preference Optimization (Rafailov et al., arXiv 2305.18290) showed it is not, for preference data. With a clean derivation, DPO turns the RLHF objective into a single classification-style loss on preference pairs, optimized directly on the policy with a frozen reference model as a regularizer. No separate reward model, no sampling loop, no critic. It is stable and cheap, and it became the default alignment step for a large fraction of open models for exactly that reason.

DPO spawned its own family, and the honest framing is that the descendants are tradeoffs, not upgrades. IPO swaps the loss to keep regularization binding on near-deterministic preferences. KTO changes the data requirement, learning from unpaired good/bad labels instead of curated pairs, which is what real feedback logs usually look like. SimPO drops the reference entirely for a length-normalized margin, halving per-step cost and reporting strong benchmark wins, with the load-bearing caveat that removing the reference removes the anchor against drift and that length-biased benchmarks flatter a method that writes longer. The right pick is set by your data shape and compute budget, not by which paper is newest, which is the side-by-side the DPO tutorial works through on real data. In practice the policy is adapted with parameter-efficient methods rather than full fine-tuning, the LoRA-family tradeoffs underneath most of these runs.

The DPO branch made preference tuning cheap and stable. It did not make the reward more trustworthy, because it is still learning from preferences, which are still a proxy for quality. For tasks where you can do better than a proxy, a third move was waiting.

The branch that lost: make the reward denser

There was another answer to reward hacking that looked more promising than either branch above, and its failure is the most instructive part of the map. If the problem is that one scalar at the end of a long chain leaves the policy room to bluff, then supervise the steps. Process reward models do exactly that, scoring each reasoning step instead of the final answer. "Let's Verify Step by Step" (Lightman et al., arXiv 2305.20050) trained a PRM on 800,000 step-level human annotations, released as PRM800K, and reported it solving 78% of problems from a representative subset of the MATH test set, beating outcome supervision at matched sampling budgets. Denser signal, real credit assignment, a reward much harder to satisfy by accident. On paper this should have been the future.

It is not the future, and DeepSeek's R1 report is unusually candid about why. Its "Unsuccessful Attempts" section rules PRMs out on three grounds: defining a fine-grained step in general reasoning is hard; deciding whether an intermediate step is correct is itself hard, since model annotation is unreliable and human annotation does not scale; and once a model-based PRM is in the loop it "inevitably leads to reward hacking," with retraining the reward model billed on top. The same section rules out MCTS-style search for a related reason, that token generation presents an exponentially larger search space than chess and the fine-grained value model that would guide the search is the part nobody knows how to train.

Read along the reward axis, this is the sharpest lesson on offer. PRMs made a learned reward finer-grained, which multiplies the surface a policy can probe rather than shrinking it: every step becomes a place to be graded, and therefore a place to be gamed. The move that won went the other direction, to a reward that is coarser but not learned at all. Sparse and checkable beat dense and learned, which is this article's argument compressed into a single comparison.

The jump that mattered: verifiable rewards

The largest capability jump in recent post-training came from changing the reward into something a policy cannot flatter. On tasks where correctness is checkable (math with a known answer, code with unit tests, anything a parser can decide) you do not need a learned reward model at all. You run the model, check the output against ground truth, and reward the verified-correct completions.

This is RLVR, RL with verifiable rewards. The name came from AI2's Tülu 3 (Lambert et al., arXiv 2411.15124, November 2024), which framed it as a simplification of RLHF rather than a new algorithm: keep the objective, replace the reward model with a verification function. DeepSeek-R1 (arXiv 2501.12948, January 2025) then made the case impossible to ignore, training R1-Zero on rule-based accuracy and format rewards only and stating the reason for avoiding neural reward models directly: at that scale they get reward-hacked. The size of the effect is what ended the argument. Over that RL run, R1-Zero's AIME 2024 pass@1 went from 15.6% to 71.0%, reaching 86.7% with majority voting over 64 samples, from a base model and a checker, with no preference data and no reward model anywhere in the loop. By 2026 RLVR is the dominant reasoning post-training paradigm.

The consequence is this article's through-line stated plainly: in RLVR the verifier is the product. The hard engineering moves from tuning the optimizer to writing the checker, which is now the real task specification. Two teams running identical GRPO code on identical base weights get different models because they wrote different verifiers.

Where "cannot be hacked" stops being true

The strong version of the RLVR claim, that a deterministic check leaves nothing to hack, is the part that has aged least well, and it is worth correcting inside the map rather than letting it stand.

A verifier is a program, and programs have bugs. Policies learn to manipulate the checker, memorize test cases, and exploit credit leakage from spurious reasoning traces (ICLR 2026, arXiv 2604.15149), and imperfect verifiers inject both false positives, which reward hackable patterns, and false negatives, which throw away correct answers and starve the run of gradient (arXiv 2510.00915). Answer-matching is the common culprit: a checker that string-matches a final expression marks an algebraically equivalent correct answer wrong.

Then there is the result that should make anyone reporting an RLVR gain nervous. "Spurious Rewards" (arXiv 2506.10947) ran RLVR on Qwen2.5-Math-7B with rewards carrying no information at all and still got most of the improvement: random rewards lifted MATH-500 by 21.4 absolute points and format-only rewards by 13.8, against 29.1 from ground-truth rewards. The reward was not teaching the model anything; RL was surfacing a behavior (thinking in code) the base model already had, and the effect largely failed to reproduce on other model families.

The honest restatement is narrower than the slogan and still worth the switch: a verifiable reward is far harder to hack than a learned one, because exploiting it means finding a bug rather than finding a preference. It is not unhackable, and the failure modes moved from the reward model into your verifier code, where you are less likely to be looking for them.

The optimizer evolution was about stability, not capability

The optimizer underneath RLVR did evolve, and it is instructive that the evolution was almost entirely about keeping long runs alive.

GRPO (introduced in DeepSeekMath, arXiv 2402.03300, and made famous by the R1 line) dropped PPO's value network for a group-relative baseline: sample a group of completions per prompt, standardize each one's reward within the group, use that as the advantage. That removes one of the four models PPO juggles and is what made single-node RLVR practical. Its known failure is structural: a group where every sample is right or every sample is wrong has zero reward variance and contributes zero gradient, so the model has to already succeed sometimes for a prompt to teach it anything. The Qwen3 technical report (arXiv 2505.09388) is a useful calibration on how much curation that implies. Its reasoning-RL stage ran GRPO on 3,995 query-verifier pairs, a set small enough to have been filtered by hand for exactly this property, sitting between a long-CoT cold-start SFT stage before it and thinking-mode fusion plus general RL after. The RL stage of a frontier open-weights model is a few thousand well-chosen prompts, not a few million scraped ones.

DAPO (arXiv 2503.14476, ByteDance Seed and Tsinghua) answered the resulting instabilities with four named fixes, the load-bearing two being Clip-Higher, which decouples the upper and lower clipping ranges to stop entropy collapse, and dynamic sampling, which filters out zero-gradient groups instead of paying for them. It reported 50 points on AIME 2024 from a Qwen2.5-32B base, the paper's own 2025-era baseline, with training code released on verl.

GSPO (arXiv 2507.18071, the Qwen team) moved importance sampling from the token level to the sequence level, on the argument that a sequence-level reward should be paired with a sequence-level ratio. Its most cited practical effect is stabilizing mixture-of-experts RL, where token-level ratios swing wildly as routing shifts between updates; Qwen credits it in the post-training of the Qwen3 and Qwen3.5 lines. It is a one-line change in current tooling: Hugging Face TRL exposes it as GRPOConfig(importance_sampling_level="sequence") on the standard GRPOTrainer.

GRPO to DAPO to GSPO is three increasingly careful answers to the same stability question, not three new capabilities. None of them raised the ceiling. The capability ceiling was set by the verifier and the base model the entire time.

The newest branch: rubric rewards for tasks with no checker

The obvious hole in RLVR is everything that is not math or code, and the 2025-2026 answer is rubric rewards. Instead of one binary check, you write a checklist of instance-specific criteria and score a response against it, usually with a model as the grader. Rubrics as Rewards (Gunjal et al., arXiv 2507.17746) is the reference formulation, reporting relative gains up to 31% on HealthBench and 7% on GPQA-Diamond over Likert-style LLM-as-judge baselines, plus the useful secondary finding that structured rubrics make small graders behave more like large ones. Those are the authors' own numbers on their own setup.

The most instructive production instance is Kimi K2's general-RL stage (arXiv 2507.20534), which pairs RLVR with a self-critique rubric reward and then does the thing the rubric papers imply but rarely spell out: the critic is refined with verifiable signals during training, using on-policy rollouts from verifiable-reward prompts to distill RLVR's objective signal into the evaluation model so its subjective judgments stay anchored to something checkable. Its rubric set includes a category the team calls prescriptive rubrics, whose stated job is to eliminate reward hacking. Both details are concessions, and they are the right ones: you re-ground a learned grader on checkable data because a learned grader drifts, and you write anti-hacking rules because you expect hacking. Budget for both up front rather than discovering them mid-run.

Read along this article's axis, though, rubric rewards are a step sideways. Decomposing a fuzzy judgment into checkable criteria is better bookkeeping than asking a judge for a 1-to-10 score, and it makes the reward auditable the way a constitution is. But the grader is still a model, so the reward is still learned and still gameable, and the 2026 work reproducing and detecting reward hacking in rubric-based RL (arXiv 2606.04923) found exactly that. My read: rubric rewards are Constitutional AI's branch with better instrumentation, not RLVR's branch extended to soft tasks. Use them where you would have used a judge model, hold them to judge-model scrutiny (the discipline the evaluation-crisis piece sets out), and never report them as if a checklist made the signal deterministic.

Agentic RL: the same argument at a longer horizon

The other live frontier is training agents rather than answers: multi-turn trajectories with tool calls, retrieval, code execution, and an outcome at the end. The reward axis holds here and gets harder in a specific way. A verifiable end-state (did the test suite pass, is the repo in the right state) is a clean RLVR-style signal, but attributing that one bit across a hundred turns is a credit-assignment problem single-turn methods never had to solve. That is why the 2026 literature is full of turn-level and step-level advantage schemes (agentic policy optimization, arXiv 2507.19849; a growing credit-assignment survey literature, arXiv 2509.02547) rather than new losses.

The frontier recipes read that way too. Kimi K3's report (arXiv 2607.24653, July 2026) describes RL run across general, agentic, and coding domains at multiple reasoning-effort levels, and the load-bearing part of that sentence is the domain list rather than the algorithm. What was built there is environments, one per capability, each with an outcome you can assert on. The optimizer is not the announcement.

The practitioner consequence is one this blog keeps arriving at from other directions: before you train an agent with RL you need an environment whose end-state you can assert on and a harness that measures reliability rather than best-of-N luck, the same requirement production agent work hits with no RL involved. If you cannot score a trajectory honestly, you cannot train on it honestly.

Self-play and the edge of the map

The far end of the current map is self-play: a model that generates its own problems, solves them, verifies the solutions, and trains on the verified-correct ones, closing the loop without a fixed external dataset. Absolute Zero and R-Zero are the reference points, and the results are real. R-Zero reports a challenger-solver co-evolution loop lifting a Qwen3-4B-Base model by 6.49 points average across math benchmarks after three iterations, on the authors' own evaluation.

Look closely at that sentence and the shape is visible: three iterations, single-digit points. Every feedback loop observed so far shows bounded improvement with diminishing returns, not the accelerating returns a true self-improvement story would require. That is the position this blog has held since the neural-symbiosis piece, and nothing in the 2026 self-play literature has moved it.

The loop is gated by two things it cannot bootstrap past. The first is the verifier: you can only self-improve on what you can check, which is why gains concentrate in math and code. The second is the base model's reachable distribution, and here the evidence is genuinely contested. Yue et al. (arXiv 2504.13837) measured RLVR-trained models against their base models at large k and found the ordering flips: RLVR wins at pass@1 by concentrating probability on paths the base model could already find, while the base model wins at pass@256. NVIDIA's ProRL (arXiv 2505.24864) is the strongest counter-evidence, reporting that with over 2,000 training steps, KL control, and reference-policy resets, a 1.5B model does find solutions inaccessible to its base under heavy sampling.

That argument is not settled and I would not write as if it were. But notice what both sides describe: a bounded expansion bought with thousands of steps and careful stabilization, not a loop that compounds. Even the optimistic result is a diminishing-returns curve with a better constant. The map ends at the verifier and the base model, every time.

The map in one table

Laid out along the reward axis, with the two columns that actually decide whether you can run a method (what it costs to hold in memory, and what data it demands) next to the one that decides whether you should trust the result.

Read the columns rather than the rows and two things fall out. The model count drops as the reward gets more honest, so the honest options are also the cheap ones. And partway down, the failure column stops describing the reward and starts describing your code.

A protocol for choosing and running a post-training method

Tracking the reward signal instead of the optimizer pays off as a decision procedure you can execute this week. Work through it in order.

1. Classify the reward before choosing anything else. Can a program decide correctness cheaply and exactly? RLVR. Is quality a preference between two responses? The DPO family. Is it a multi-criteria judgment a person could grade with a checklist? Rubric rewards, with the caveats above. This one classification decides more than every subsequent choice combined.
2. Write the verifier first, and attack it. In RLVR territory the verifier is the deliverable. Before training, run it over known-correct and known-wrong answers, including equivalent forms, alternate formats, and deliberate near-misses. Count its false positives and false negatives: that number is your ceiling. A labeled adversarial set of a few dozen cases, run before any GPU time, catches most of it:

```python # The verifier is the task spec, so test it like one, before the first run. from math_verify import parse, verify # pip install math-verify

def check(gold: str, pred: str) -> bool: return verify(parse(gold), parse(pred))

# gold, model output, what a correct verifier must return CASES = [ ("1/2", "0.5", True), # equivalent forms ("(x+1)^2", "x^2+2x+1", True), # algebraic equivalence ("{1,3} \\cup {2,4}", "{1,2,3,4}", True), # set notation ("12", "12 apples", True), # units in the response ("12", "The answer is 12.", True), # answer wrapped in prose ("12", "1.2", False), # near miss ("12", "12 or maybe 13", False), # hedged, should not count ]

fp = sum(1 for g, p, want in CASES if check(g, p) and not want) fn = sum(1 for g, p, want in CASES if not check(g, p) and want) print(f"false positives {fp}, false negatives {fn}, of {len(CASES)}") ```

Run it rather than assuming it. A stock parser will disagree with your intent on some of these, and each disagreement is either free reward you are about to hand the policy or a correct answer you are about to punish.

1. Treat the optimizer as near-interchangeable plumbing. Start with GRPO. Add DAPO's Clip-Higher and dynamic sampling when you see entropy collapse or many zero-variance groups. Switch to sequence-level importance sampling (GSPO) for a mixture-of-experts policy or long trajectories. None of it fixes a bad reward.
2. Check that your prompts are in the learnable band. Sample the base model several times per prompt and record the pass rate. Always-right and always-wrong prompts contribute no gradient under a group-relative baseline, so curating for partial success usually beats any optimizer change.
3. Pick the baseline before the run, not after. The comparison is your SFT checkpoint on held-out data, evaluated identically. Never report the training reward: it is the most contaminated number in the run, and it rises even when the model is exploiting your checker.
4. Run a spurious-reward control. Repeat a short run with a random or format-only reward. If it recovers most of your headline gain, the reward is not what is teaching the model, and you have learned something worth more than the gain.
5. Validate on a second base model family, and report pass@1 with pass@k. RLVR results repeatedly fail to transfer across families, so one family is an anecdote about that family. And if the gain at k=1 comes with a loss at large k, you sharpened the distribution rather than expanding it, which is often fine for production and should be stated rather than hidden.

If you find yourself reaching for a full PPO-with-reward-model setup, be sure you need it, because the field spent five years building cheaper paths around it for good reasons. The optimizer is rarely where the win lives. The win lives in the signal, and increasingly in how honestly you measure it.

Key Takeaways

1. Track the reward signal, not the optimizer. Every major RL post-training jump changed where the reward comes from. Read along the optimizer axis instead and coherent progress looks like acronym churn.
2. RLHF's weakness defines the field. A learned reward model is a proxy the policy optimizes against, so it gets reward-hacked (length, sycophancy), and PPO's four-model setup is expensive to run well. Everything downstream answers one of those two problems.
3. Constitutional AI scaled the labels, not the honesty. RLAIF swaps human preferences for AI-generated ones against written principles, and the 2026 versions (Anthropic's tiered constitution, OpenAI's deliberative alignment) express those principles far better. The reward underneath is still a learned, gameable proxy.
4. DPO deleted the reward model for preference data. One classification-style loss, frozen reference, no critic, no sampling loop. Its descendants (IPO, KTO, SimPO) change the loss, the data requirement, or drop the reference: tradeoffs, not upgrades, and reference-free wins can come partly from length bias.
5. Sparse and checkable beat dense and learned. Process reward models made the learned signal finer-grained and lost anyway, because every graded step is another place to be gamed. RLVR was the capability jump because it changed the reward, not the granularity: named in Tülu 3, made unavoidable by DeepSeek-R1-Zero going from 15.6% to 71.0% pass@1 on AIME 2024 on rule-based rewards alone, it moved the hard engineering from the optimizer to the verifier, which is now the real task specification.
6. "Nothing to reward-hack" is too strong. Policies exploit verifier bugs and memorize test cases, imperfect checkers inject false positives and negatives, and random rewards recovered 21.4 of 29.1 points on one model family. Verifiable rewards are much harder to hack, not unhackable, and the bugs now live in your code.
7. GRPO, DAPO, and GSPO are stability fixes. Group-relative baselines, Clip-Higher plus dynamic sampling, and sequence-level importance sampling answer one stability question. The capability ceiling stays set by the verifier and the base model.
8. Rubric rewards extend Constitutional AI's branch, not RLVR's. Checklist grading beats a judge model's Likert score as bookkeeping and reports real gains, but the grader is still a model, so the reward is still learned and still hackable.
9. Self-improvement is bounded. Every observed loop shows diminishing returns, gated by the verifier and the base model's reachable distribution. Whether RL expands that distribution is contested (pass@k evidence against, ProRL for), but both sides describe a bounded expansion, not a compounding one.

The Acing AI newsletter reads new training methods for what they change about the reward, not the marketing. Subscribe for the map, not the hype.