Preference Optimization After DPO: IPO, KTO, and SimPO Compared

Preference optimization is the cheap, stable half of post-training that most teams reach for before they touch reinforcement learning, and the menu has grown past DPO into a family of objectives (IPO, KTO, SimPO) that each claim to fix something. The marketing reads like a strict upgrade path. The reality is a set of tradeoffs, and the cleanest way to see them is to run all four with the same model, the same adapter, and as close to the same data as each objective allows, changing as little as possible between runs. This tutorial does that, then reads the differences honestly: what each objective changes, where the reference-free methods genuinely win, and why a benchmark victory on AlpacaEval does not always survive contact with your task.

DPO and its descendants sit one rung below RLVR in complexity and one rung above plain supervised fine-tuning. They learn from preference pairs (a chosen response and a rejected one) without a reward model and without sampling rollouts, which is exactly why they are the default starting point for alignment. The RLHF-to-RLVR lineage gives the context; this piece is the runnable craft.

The four objectives in one paragraph each

DPO (Direct Preference Optimization) is the anchor. It skips the reward-model-then-PPO pipeline of classic RLHF and optimizes the policy directly from preference pairs, using a frozen reference model (your SFT checkpoint) to regularize: the loss rewards making the chosen response more likely and the rejected one less likely than the reference does, with a beta term controlling how far the policy may drift. DPO is stable, well understood, and the method everything else is measured against.

IPO (Identity Preference Optimization), from Azar et al. at DeepMind, fixes a specific DPO failure: when preferences are nearly deterministic (the chosen response is almost always preferred), DPO's objective can overfit and push the policy to ignore the reference regularization entirely, collapsing toward putting all mass on chosen. IPO replaces the log-sigmoid loss with a squared term that keeps the regularization binding regardless of how lopsided the preferences are. It is the conservative choice when your preference data is clean and strong.

KTO (Kahneman-Tversky Optimization), from Ethayarajh et al., changes the data requirement, not just the loss. DPO and IPO need paired data (the same prompt with a chosen and a rejected completion). KTO needs only a binary label per example: this output is good, that one is bad, with no pairing. That is a large practical difference, because unpaired thumbs-up and thumbs-down signals are far easier to collect at scale than curated pairs. The loss is derived from prospect theory, weighting gains and losses asymmetrically the way human utility does.

SimPO (Simple Preference Optimization), from Meng et al. at Princeton (NeurIPS 2024), drops the reference model entirely. DPO's reward is the policy's log-probability ratio against the reference; SimPO uses the average log-probability of the sequence directly, length-normalized, with a target reward margin. Removing the reference halves the memory and compute (no second forward pass through a frozen model) and aligns the training reward with the quantity used at inference. SimPO reports beating DPO across AlpacaEval 2, Arena-Hard, and MT-Bench. The catch is in the takeaways below.

Setup

Pin the stack. DPO and IPO share DPOTrainer and differ by a single loss_type argument; SimPO ships in TRL's CPOTrainer (the reference-free family, currently under trl.experimental); KTO has a dedicated KTOTrainer. Where TRL files each objective is itself a map of what the objective removes.

# Python 3.11+, CUDA 12.x, one 24GB GPU is enough with LoRA and the settings below.
# Versions current as of July 2026.
pip install \
  "trl==1.9.2" \
  "transformers==5.14.1" \
  "peft==0.19.1" \
  "datasets==5.0.0" \
  "accelerate==1.14.0" \
  "torch==2.13.0"

We align Qwen3.5-4B, the post-trained 4B of the current Qwen small series, on a standard preference set. The same LoRA adapter approach used everywhere else applies: preference optimization shifts behavior, not knowledge, so a rank-16 adapter captures it cheaply and keeps the base weights intact.

from datasets import load_dataset

# UltraFeedback Binarized: prompt + chosen + rejected, the standard DPO-style format.
dataset = load_dataset("trl-lib/ultrafeedback_binarized", split="train")

The shared trainer

Here is the part that makes the comparison fair: for DPO and IPO it is literally one trainer, one config, one argument that selects the objective. Everything else is held constant.

from trl import DPOConfig, DPOTrainer
from peft import LoraConfig

def train(loss_type, beta):
    """Train one DPOTrainer objective: "sigmoid" is DPO, "ipo" is IPO.
    Everything else is identical across runs so the comparison is honest."""
    config = DPOConfig(
        output_dir=f"qwen35-4b-{loss_type}",
        loss_type=loss_type,
        beta=beta,                  # regularization strength against the reference
        learning_rate=5e-6,
        per_device_train_batch_size=4,
        gradient_accumulation_steps=4,
        max_length=1024,
        max_prompt_length=512,
        num_train_epochs=1,
        bf16=True,
        logging_steps=10,
    )
    trainer = DPOTrainer(
        model="Qwen/Qwen3.5-4B",
        args=config,
        train_dataset=dataset,
        peft_config=LoraConfig(r=16, lora_alpha=32, task_type="CAUSAL_LM"),
    )
    trainer.train()
    return trainer

# DPO: the anchor. Frozen reference model, log-sigmoid loss.
train(loss_type="sigmoid", beta=0.1)

# IPO: squared loss that keeps regularization binding on near-deterministic prefs.
train(loss_type="ipo", beta=0.1)

SimPO does not live in DPOTrainer, and the reason is the method itself: DPOTrainer is built around the frozen reference model, which is exactly the thing SimPO deletes. TRL files it with the reference-free CPO family instead. Same preference pairs, same adapter, same learning rate; the config swaps the reference-anchored loss for a length-normalized reward with a target margin.

# SimPO: reference-free. No second forward pass through a frozen model.
from trl.experimental.cpo import CPOConfig, CPOTrainer

simpo_config = CPOConfig(
    output_dir="qwen35-4b-simpo",
    loss_type="simpo",
    cpo_alpha=0.0,          # 0.0 = pure SimPO; nonzero blends in a CPO SFT term
    beta=2.0,               # scales the reward; SimPO runs far hotter than DPO's 0.1
    simpo_gamma=0.5,        # the target reward margin
    learning_rate=5e-6,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    max_length=1024,
    max_prompt_length=512,
    num_train_epochs=1,
    bf16=True,
    logging_steps=10,
)
simpo_trainer = CPOTrainer(
    model="Qwen/Qwen3.5-4B",
    args=simpo_config,
    train_dataset=dataset,
    peft_config=LoraConfig(r=16, lora_alpha=32, task_type="CAUSAL_LM"),
)
simpo_trainer.train()

KTO changes the data shape itself (unpaired binary labels), so it gets a dedicated KTOTrainer. The point of holding the model, data, adapter, and learning rate constant across DPO, IPO, and SimPO is that the objective is genuinely the only thing changing; any difference in the result is the loss function (and, for SimPO, the absent reference model), not a confound.

# KTO needs unpaired {prompt, completion, label: bool} rows, not chosen/rejected pairs.
from trl import KTOConfig, KTOTrainer

kto_data = load_dataset("trl-lib/kto-mix-14k", split="train")
kto_trainer = KTOTrainer(
    model="Qwen/Qwen3.5-4B",
    args=KTOConfig(output_dir="qwen35-4b-kto", beta=0.1, bf16=True),
    train_dataset=kto_data,
    peft_config=LoraConfig(r=16, lora_alpha=32, task_type="CAUSAL_LM"),
)
kto_trainer.train()

Reading the results honestly

Run these and you will get four aligned models. The temptation is to rank them by one number (AlpacaEval win rate is the usual one) and crown a winner. Resist it, because the evaluation crisis lesson applies sharply to preference optimization, where the standard benchmarks have a known bias that the methods can exploit.

Here is the trap. SimPO's length-normalized reward and AlpacaEval's documented length bias point the same direction: longer answers score higher on the benchmark, and a reward that touches length can learn to produce them. A method can win AlpacaEval partly by writing more, which is not the same as being better. The right read is not "which method scored highest" but "which method improved the behavior I care about, measured on my distribution, controlling for length." The concrete procedure: generate from each of the four checkpoints on the AlpacaEval 2 prompt set and report the length-controlled (LC) win rate against your starting checkpoint's generations, not against the leaderboard reference. The LC variant exists precisely because the raw metric was gameable by verbosity, and your SFT starting point, not GPT-4's outputs, is the baseline that answers whether the alignment run earned its cost.

Pair that with a check the judge cannot bias. A win rate is still an LLM grading an LLM, so run each of the four checkpoints against the same SFT baseline on a scored public benchmark with a deterministic answer key: IFEval for instruction-following, GSM8K for math, HumanEval for code, whichever is closest to your task. Report the before/after accuracy alongside the LC win rate. This is the measurement that catches the failure the previous section predicts, because forgetting from a reference-free run shows up as a capability number falling while the preference win rate rises, and only the pair of numbers makes that visible.

The reference-free tradeoff is the other thing the benchmark hides. Dropping the reference model (SimPO) and dropping the pairing requirement (KTO) both save real resources, and on clean academic data they match or beat DPO. But the reference model in DPO is not just overhead, it is an anchor: it is what keeps the policy from drifting too far from a known-good distribution. Remove it and you remove a stabilizer. On noisy or out-of-distribution preference data, reference-free methods are more prone to catastrophic forgetting and reward over-optimization, because nothing is holding the policy back. The win on a curated benchmark does not guarantee the win on your messier production data.

Which to actually use

The decision is not "the newest one." It is a match between the objective's assumption and your data.

• Start with DPO. It is the stable, understood default. If it gives you the behavior you want, you are done, and the descendants are solving problems you do not have.
• Reach for IPO when your preferences are strong and clean and you see DPO overfitting (the policy collapsing onto chosen, the reference regularization stops binding). IPO's squared loss is the targeted fix for exactly that.
• Reach for KTO when your data is unpaired. If what you actually have is thumbs-up and thumbs-down signals rather than curated pairs, KTO uses them directly instead of forcing you to construct pairs you do not have. This is often the real-world situation, and it is KTO's strongest argument.
• Reach for SimPO when memory or throughput is the constraint and your preference data is clean. The reference-free design genuinely halves the per-step cost, which matters at scale. Validate harder for length inflation and forgetting, because you have removed the anchor that would have caught them.

None of these is a strict upgrade over DPO. Each trades something (a stabilizer, a data assumption, an inductive bias) for something else. The honest framing, the one the benchmark leaderboards bury, is that preference optimization is a small family of objectives with different failure boundaries, and the right choice is set by the shape of your data and your compute budget, not by which paper is most recent.

Key Takeaways

1. TRL's layout mirrors the method map. DPO and IPO are a one-argument loss_type swap inside DPOTrainer; reference-free SimPO lives with the CPO family; unpaired KTO gets its own trainer. An honest comparison still holds model, data, and adapter constant and changes only the objective.
2. DPO is the anchor; the rest are targeted fixes. IPO fixes overfitting on near-deterministic preferences with a squared loss; KTO drops the pairing requirement; SimPO drops the reference model. None is a blanket upgrade.
3. KTO changes the data requirement, not just the loss. It learns from unpaired binary good/bad labels, which are far easier to collect at scale than curated chosen/rejected pairs. That practical difference is often the deciding factor.
4. SimPO's reference-free design halves per-step cost. No second forward pass through a frozen model, and the training reward matches the inference-time quantity. The savings are real and matter at scale.
5. Reference-free removes a stabilizer. The reference model anchors the policy to a known-good distribution. Without it, SimPO and KTO are more prone to forgetting and over-optimization on noisy or out-of-distribution data.
6. AlpacaEval length bias and SimPO's length normalization point the same way. A method can win the benchmark partly by writing longer answers. Report win rates length-controlled, against your SFT baseline, on your distribution, and pair them with a deterministic benchmark (IFEval, GSM8K, HumanEval) so a capability regression cannot hide behind a rising win rate.
7. Pick by data shape and budget, not recency. Start with DPO; use IPO for clean strong preferences, KTO for unpaired data, SimPO when compute is the binding constraint. The newest objective is not automatically the right one.

The Acing AI newsletter covers alignment the way this tutorial does: the objective that fits your data, not the one that topped last month's leaderboard. Subscribe for the comparison nobody runs.