RLVR in Practice: Build a Verifiable-Reward Loop with GRPO

RLVR is the post-training paradigm behind every strong open reasoning model shipped since DeepSeek-R1, and the part that does the work is not the policy-gradient math, it is the verifier. Reinforcement learning with verifiable rewards replaces a learned, gameable reward model with a deterministic checker: did the math answer match, did the code pass the tests, did the output parse. Get the verifier right and the optimizer is almost an implementation detail. Get it wrong and no amount of GRPO, DAPO, or GSPO tuning will save you, because the model will optimize exactly the loophole you left open.

This tutorial builds a working RLVR loop on a small reasoning task, end to end, with a real verifier rather than a placeholder. We use GRPO because it is the algorithm that made this approach practical, then explain what DAPO and GSPO change and when you actually need them. The framing throughout is the one that matters in production: the reward function is the specification of the task, and writing it is the hard part.

Why verifiable rewards changed post-training

Classic RLHF trains a reward model on human preference pairs, then optimizes the policy against that learned reward with PPO. It works for taste (helpfulness, tone, formatting) but it has a structural weakness: the reward model is itself a neural network, so it can be gamed. Push the policy hard enough and it finds adversarial inputs that score high and read as nonsense. This is reward hacking, and it is why RLHF runs need a KL leash and careful early stopping.

RLVR sidesteps the whole problem for any task with a checkable answer. Instead of a learned reward, you score a generation with a program: a math-answer comparison, a unit-test run, a regex, a compiler. The reward is the ground truth, not an approximation of it, so there is nothing to hack. That is why the first big RLVR wins were in math and code, where verification is cheap and exact, and why the technique does not transfer cleanly to open-ended writing, where it is not. This is the same generator-plus-incorruptible-verifier loop that the neural symbiosis piece argues is the engine of recent gains, and it carries the same caveat: the improvement is bounded by what your verifier can actually check.

The honest baseline question comes first. Before you reach for RL, ask whether supervised fine-tuning on correct traces gets you most of the way. It usually gets you a lot of the way. RLVR earns its complexity when the model can already produce correct answers sometimes and you want to raise the rate it does so reliably, which is exactly the regime where a verifier can separate good rollouts from bad ones at scale.

How GRPO works, briefly

GRPO (Group Relative Policy Optimization), introduced in the DeepSeekMath paper and made famous by DeepSeek-R1, is the algorithm that stripped PPO down for this setting. PPO needs a separate value network (a critic) to estimate the advantage of each action. GRPO throws the critic away. For each prompt it samples a group of G completions, scores all of them with the verifier, and computes each completion's advantage as its reward standardized within the group: subtract the group mean, divide by the group standard deviation. A completion that beats its siblings gets a positive advantage and is reinforced; one that loses gets a negative advantage and is suppressed.

That single change, group-relative advantage instead of a learned critic, is what makes RLVR cheap enough to run on a single node. There is no value model to train, the baseline is just the other samples in the group, and the verifier supplies the only signal. The cost shows up elsewhere: you generate G completions per prompt (8 to 16 is typical), so rollout generation dominates wall-clock time, which is why every serious setup pairs GRPO with a fast inference engine like vLLM for the sampling step.

Setup

Pin the stack. RLVR tooling moves fast and the trainer APIs have churned, so exact versions matter more here than in most tutorials.

# Python 3.11+. CUDA 12.x. Tested on a single 48GB GPU (one 24GB card works with the
# batch settings noted inline). Versions current as of July 2026.
pip install \
  "trl==1.8.0" \
  "transformers==5.14.1" \
  "peft==0.19.1" \
  "datasets==5.0.0" \
  "accelerate==1.14.0" \
  "vllm==0.25.1" \
  "math-verify==0.9.0"
# No explicit torch pin: vLLM ships against a specific torch build (2.13.x here),
# so let it resolve torch rather than fighting it.

We train Qwen3.5-4B as the policy, the post-trained 4B of the current Qwen small series (the 3.5 line drops the -Instruct suffix; the pretrained-only variant is Qwen3.5-4B-Base, and the vision tower it ships with simply goes unused on a text-only task). It is small enough to fit on one card with LoRA, current enough to be a fair example, and already competent at arithmetic reasoning, which is the precondition RLVR needs: the model has to succeed sometimes for the group-relative signal to have any variance to learn from. A model that gets every rollout wrong gives every completion the same zero reward, a zero-variance group, and therefore zero gradient.

We use LoRA for the policy update. Full-parameter RLVR is possible but the LoRA default holds here too: the adapter captures the behavioral shift RLVR induces at a fraction of the memory, and it keeps the base weights frozen so a bad run is cheap to discard.

Step 1: the dataset

RLVR needs prompts paired with checkable ground-truth answers, not full reasoning traces. The model generates the reasoning itself; you only need to know the right final answer to score it. We use a standard grade-school-to-competition math set.

from datasets import load_dataset

# Each row: `prompt` (the problem, already in conversational format) and `solution`
# (the verified final answer, for the verifier). We do NOT use any reference solution
# text. RLVR scores the model's OWN generated answer against `solution`.
dataset = load_dataset("trl-lib/DeepMath-103K", split="train").select(range(4000))

SYSTEM = (
    "Solve the problem step by step. "
    "Put your final answer inside \\boxed{}."
)

def add_system(example):
    return {"prompt": [{"role": "system", "content": SYSTEM}] + example["prompt"]}

dataset = dataset.map(add_system)

The \boxed{} instruction is not cosmetic. It gives the verifier a reliable place to find the answer, which separates the easy parsing problem from the hard correctness problem. A surprising share of "the model is wrong" failures in early RLVR runs are actually "the verifier could not find the answer" failures, and they silently cap your reward at well below what the model deserves.

Step 2: the verifier (the part that matters)

This is the heart of the system. The reward function receives the model's completions and the ground-truth answers, and returns a scalar reward per completion. We use math-verify, which parses mathematical expressions and checks equivalence symbolically, so 1/2, 0.5, and \frac{1}{2} all compare equal. String matching would mark two of those three wrong.

from math_verify import parse, verify

def correctness_reward(completions, solution, **kwargs):
    """+1.0 if the model's boxed answer is mathematically equivalent to ground truth,
    else 0.0. This is the ground-truth signal: deterministic, not learned, unhackable
    within the limits of what `verify` can check."""
    rewards = []
    for completion, gold in zip(completions, solution):
        text = completion[0]["content"]
        gold_parsed = parse(gold)
        pred_parsed = parse(text)  # pulls the \boxed{} expression out of the text
        if not gold_parsed or not pred_parsed:
            rewards.append(0.0)      # unparseable: treat as wrong, but see the note below
            continue
        rewards.append(1.0 if verify(gold_parsed, pred_parsed) else 0.0)
    return rewards

def format_reward(completions, **kwargs):
    """Small shaping reward for producing a single \\boxed{} answer at all.
    Kept an order of magnitude below correctness so it never dominates."""
    import re
    pattern = re.compile(r"\\boxed\{.+?\}")
    return [0.1 if pattern.search(c[0]["content"]) else 0.0 for c in completions]

Three points define the craft here, and they are where real runs succeed or fail.

First, the unparseable case is a decision, not a default. Returning 0.0 for an answer the parser cannot read teaches the model that ill-formatted answers are as bad as wrong ones, which is usually what you want. But if your parser is brittle, you are punishing correct answers for cosmetic reasons and starving the run of signal. Log your parse-failure rate separately from your wrong-answer rate; if the former is more than a few percent, fix the verifier before you touch any hyperparameter.

Second, shaping rewards are a loaded gun. The format_reward is deliberately ten times smaller than the correctness reward. Make a shaping term too large and the model optimizes it instead of the task: a length bonus produces padding, a format bonus produces well-formatted wrong answers. Every reward term you add is a thing the model will exploit if you let it.

Third, the verifier defines the task. If math-verify accepts an answer your downstream use would reject, the model will learn to produce exactly that. The reward function is the specification. Read it as adversarially as the model will.

Step 3: the GRPO loop

With the dataset and verifier in place, the trainer is short. The interesting parameters are the group size, the generation settings, and the KL coefficient.

from trl import GRPOConfig, GRPOTrainer
from peft import LoraConfig

config = GRPOConfig(
    output_dir="qwen35-4b-rlvr",
    learning_rate=1e-6,            # RL is sensitive; 1e-6 to 5e-6, far below SFT
    per_device_train_batch_size=8,
    gradient_accumulation_steps=4,
    num_generations=8,            # G: completions per prompt. The group size.
    max_completion_length=1024,
    max_prompt_length=512,
    temperature=1.0,              # need diversity in the group, so sample hot
    beta=0.0,                     # KL coefficient. 0.0 follows DAPO; see below.
    num_train_epochs=1,
    bf16=True,
    use_vllm=True,                # offload rollout generation to vLLM, the bottleneck
    vllm_mode="colocate",
    logging_steps=1,
)

trainer = GRPOTrainer(
    model="Qwen/Qwen3.5-4B",
    reward_funcs=[correctness_reward, format_reward],  # summed per completion
    args=config,
    train_dataset=dataset,
    peft_config=LoraConfig(r=32, lora_alpha=64, task_type="CAUSAL_LM"),
)

trainer.train()

Passing a list to reward_funcs makes the trainer sum the rewards per completion (use reward_weights in the config to weight them), so the correctness signal and the format shaping combine cleanly. The single most important number to watch during training is not the loss, which is nearly meaningless in policy-gradient RL, but the mean reward and the fraction of groups with nonzero variance. If reward climbs steadily, the loop is working. If the variance fraction collapses toward zero, every group is all-right or all-wrong and there is no gradient left, which means your task is too easy or too hard for the current policy.

Where GRPO breaks, and what DAPO and GSPO fix

GRPO is the baseline, not the final word. Two well-documented failure modes drove the refinements you will see named in 2026 papers, and knowing which problem each one solves tells you whether you need it.

DAPO (Decoupled Clip and Dynamic Sampling Policy Optimization), from ByteDance and Tsinghua, targets two GRPO pathologies. Entropy collapse, where the policy becomes overconfident and stops exploring, is countered by Clip-Higher, which decouples the upper and lower PPO clipping bounds to keep low-probability tokens alive. Wasted compute on zero-gradient groups is countered by dynamic sampling, which filters out prompts where every completion is right or every one is wrong before the gradient step, since those contribute nothing. DAPO also drops the KL penalty entirely (beta=0.0 above), arguing that for reasoning tasks the policy is supposed to move far from the base model, so leashing it to the reference is counterproductive. If your runs plateau early or burn time on degenerate groups, DAPO is the upgrade.

GSPO (Group Sequence Policy Optimization), from the Qwen team (arXiv 2507.18071), targets a subtler instability: GRPO applies importance-sampling correction at the token level, and the per-token ratios accumulate high variance over long completions, which can spike into outright collapse, especially when training mixture-of-experts models where routing adds noise. GSPO moves the importance ratio and the clipping to the sequence level, matching the unit of reward (you reward whole sequences, so you should weight whole sequences). The Qwen team credits it with stabilizing the RL stage of recent Qwen3 models and with making mixture-of-experts RL training tractable. If you are doing long-horizon reasoning RL or training an MoE policy and seeing variance blow up, GSPO is the fix.

The pattern across all three is worth naming, because it is easy to read the paper sequence as escalating capability. It is not. GRPO, DAPO, and GSPO are increasingly careful answers to the same stability question, not three different capabilities. None of them changes what RLVR can learn; they change how reliably you can train it without the run diverging. The capability ceiling is set by the verifier and the base model, every time.

What this does not do

RLVR raises the rate at which a model produces verifiably correct answers. It does not teach knowledge the base model lacks, it does not generalize past what the verifier checks, and it does not help on tasks with no checkable answer. Evaluate the result the way the evaluation crisis demands: not on the training reward, which is the thing you optimized and therefore the most contaminated number you have, but on a held-out set with a baseline (the SFT model you started from) and a distribution (accuracy across difficulty bands, not one aggregate). Concretely: hold out a DeepMath slice the run never saw, add one public benchmark it never touched (MATH-500 is the standard choice at this difficulty), and report accuracy for the starting checkpoint and the RLVR checkpoint side by side, per difficulty band, scored with the same math-verify verifier you trained against. The honest question is whether the RLVR model beats the SFT model on problems neither has seen, by enough to justify the rollout compute. Often it does. Sometimes the gap is inside the noise, and the SFT baseline was the answer all along.

Key Takeaways

1. The verifier is the product, not the optimizer. RLVR replaces a gameable learned reward with a deterministic checker. Writing the reward function is the hard part; GRPO, DAPO, and GSPO are interchangeable plumbing by comparison.
2. RLVR only works where verification is cheap and exact. Math and code are the home turf because answers are checkable. It does not transfer to open-ended generation, where there is nothing incorruptible to check against.
3. GRPO drops the critic for a group-relative baseline. Sample G completions, standardize each one's reward within the group, reinforce the winners. No value network, which is what makes single-node RLVR practical.
4. The model must already succeed sometimes. A zero-variance group (all right or all wrong) produces zero gradient. Watch the nonzero-variance fraction, not the loss, which is near-meaningless in policy-gradient RL.
5. Separate parse failures from wrong answers. A brittle verifier punishes correct answers for formatting and silently caps your reward. Log the two failure rates apart and fix parsing before tuning anything else.
6. Shaping rewards get exploited. Keep format and length bonuses an order of magnitude below the correctness signal, or the model optimizes the bonus and ships well-formatted nonsense.
7. DAPO and GSPO are stability fixes, not capability upgrades. DAPO fights entropy collapse and wasted zero-gradient groups (Clip-Higher, dynamic sampling, no KL); GSPO moves importance sampling to the sequence level to stabilize long-horizon and MoE training. Neither raises the capability ceiling, which the verifier sets.
8. Benchmark against the SFT baseline, not the training reward. The training reward is the most contaminated number in the run. The real result is held-out accuracy over a distribution, measured against the supervised model you started from.

The Acing AI newsletter covers post-training the way this tutorial does: the verifier and the eval, not the leaderboard. Subscribe for the working version, not the demo.