preference-optimization
Align a fine-tuned model with preference data using DPO, ORPO, KTO, or SimPO. Use when preference pairs or thumbs-up/down feedback exist, when choosing between preference-optimization methods, or when a DPO run needs hyperparameters or debugging.
Install
npx skills add https://github.com/wshobson/agents/tree/main/plugins/llm-finetuning/skills/preference-optimization
claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install wshobson-agents@llmmart
git clone https://github.com/wshobson/agents.git
The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole wshobson/agents collection as a plugin from our marketplace. Git is the plain clone.
Skill manifest
Preference Optimization
This skill assumes finetuning-method-selection
already routed here because the data shape is
preference pairs or unpaired thumbs-up/down
feedback, not demonstrations (that's
lora-qlora-recipes) or a verifiable reward
signal (that's grpo-rlvr-training). What
follows is method selection among the DPO family,
the evidence for how much that selection actually
matters, the production training pattern, and how
to build the pairs in the first place.
Input: a routing decision (preference
optimization) plus preference pairs or unpaired
feedback, usually from an SFT checkpoint.
Output format: a validated method choice plus
a config — the kwarg values in
references/method-configs.md, not free-form
advice — that llm-finetuning-training-engineer
consumes directly.
Method Selection
| Data shape | Method | Key parameters |
|---|---|---|
| Preference pairs, default case | DPO | β=0.1, LR 5e-7–1e-6, 1–2 epochs |
| Memory-bound or no SFT checkpoint | ORPO | reference-free, fused SFT+preference in one loss |
| Unpaired thumbs-up/down | KTO | binary label per example, no pairing needed |
| Length bias observed, sweep budget available | SimPO | reference-free; see sweep grid below |
- DPO is the safe default. Use β=0.1 and a learning rate of 5e-7 to 1e-6 for 1–2 epochs. This LR is lower than the SFT LR that produced the checkpoint being aligned — porting an SFT- scale LR into a DPO run is the most common misconfiguration here, not an edge case.
- ORPO routes in when memory is the constraint, or when there's no separate SFT checkpoint to start from — it's reference-free and fuses the SFT and preference objectives into one loss, skipping the separate SFT pass and the reference-model memory cost DPO carries.
- KTO routes in when feedback is unpaired binary signal (thumbs-up/down) rather than matched preference pairs — don't force unpaired feedback into synthetic pairs to use DPO instead.
- SimPO fixes DPO's length bias but only pays off with disciplined sweeping — its published gains are a ceiling reported under a tuned sweep, not a baseline any single config will reproduce. Route here only when there's sweep budget; use DPO instead if there isn't.
- Classic RLHF (reward model + PPO) is retired outside frontier labs. Don't reach for it in a production pipeline — every method above is cheaper and better-supported for the same data shapes.
Worked Examples
- "We have an SFT checkpoint and clean paired preference data, no length-bias complaints yet." → default case → DPO at β=0.1.
- "Reviewers click thumbs-up/down per response; nothing is paired." → unpaired signal → KTO, not DPO — don't synthesize pairs to force DPO onto unpaired data.
- "GPU budget doesn't cover a separate SFT pass plus a DPO reference model." → memory-bound, no separate checkpoint → ORPO.
- "DPO output favors longer answers regardless of quality, and there's time to run a sweep." → length bias plus sweep budget → SimPO. Skip it if the sweep budget isn't actually there.
The Low-Leverage Truth
A 2026 240-H100-run study (arXiv 2603.19335) is the load-bearing evidence behind the table above: loss-function choice is worth roughly 1 percentage point of leverage, model scale is worth roughly 50. Zero of 20 DPO variants tested beat vanilla DPO. Rankings also invert with scale — a variant that wins in a small pilot can lose at deployment size.
Two practical consequences:
- Don't spend a routing decision agonizing over DPO-variant bake-offs. The table above is sufficient; deeper variant selection is low-leverage compared to data quality and scale.
- Validate at deployment scale before trusting a ranking. A method comparison run on a small pilot model doesn't transfer to the production size class — re-check the winner once scale changes.
This is also why the Method Selection table above is deliberately short: it encodes the ~1pp lever, not a ranking of DPO variants that the same study shows doesn't hold up across scale. Treat any variant-selection advice that isn't in that table — including advice that claims a specific variant "wins" — as unproven until it's been validated at the target deployment size.
Production Pattern: Iterative On-Policy DPO
A single offline DPO pass on a static preference dataset is a starting point, not the production pattern. The policy drifts away from the distribution the pairs were sampled from as training proceeds, and a static dataset goes stale against that drift. Production pipelines run DPO iteratively and on-policy instead:
- Sample completions from the current policy checkpoint.
- Score or rank the completions (reward model, judge, or task grader).
- Run a DPO pass using the current checkpoint as the reference model.
- The resulting checkpoint becomes both the new policy and the new reference for the next round.
Repeat. Each round's reference model is the prior round's output, not a fixed initial checkpoint — that's what keeps the preference signal on-policy instead of scoring against an increasingly stale distribution.
A single-pass DPO run is still a reasonable first iteration — it just isn't the whole pipeline. Plan for at least one more round once the first checkpoint exists, rather than treating pass one as the finished artifact.
Pair Construction
Build DPO/ORPO pairs from same-task passing-vs-failing trajectories — two attempts at the same underlying task, not unrelated best-and-worst examples pulled from different tasks. Within that trajectory set, select the rejected member at μ−2σ of the reward distribution, never the minimum. Naive best-vs-worst pair construction (max reward vs. absolute minimum) degrades as scale increases; the μ−2σ selection is more robust to the same scale sensitivity the low-leverage study surfaced above.
sorted_by_reward = sort(trajectories, key=reward)
chosen = sorted_by_reward[-1] # highest reward
mu, sigma = mean(rewards), stdev(rewards)
rejected = closest(sorted_by_reward, mu - 2 * sigma)
# NOT sorted_by_reward[0] — the absolute minimum
# is the naive best-vs-worst construction that
# degrades as scale increases.
For the mechanics of turning graded traces into
these pairs — including rejection sampling and
judge-scored delta selection — see
trace-to-training-data.
References
Complete TRL config blocks per method —
DPOConfig, ORPOConfig, KTOConfig, and the
SimPO sweep grid — plus Unsloth wrappers and a
catastrophic-forgetting note live in
references/method-configs.md. Those configs use
the same current-TRL API conventions established
in lora-qlora-recipes's
references/unsloth-trl-mapping.md
(processing_class, not tokenizer=).
references/method-configs.md also carries the
catastrophic-forgetting note: a too-high learning
rate is the usual cause when a preference-tuned
checkpoint loses general capability, and the fix
is almost always to drop the LR toward the low end
of the range in the Method Selection table above
before reaching for any other remediation.
Related skills: finetuning-method-selection
routes here once preference pairs or unpaired
feedback exist; lora-qlora-recipes produces the
SFT checkpoint DPO/KTO/SimPO align (ORPO's
fused path can skip it); trace-to-training-data
converts passing/failing trajectories into the
pairs this skill's Pair Construction section
consumes.
Files (agents)
-
references
-
method-configs.md 6.4 KB
Last verified: 2026-07-13 # Preference Optimization Method Configs Complete TRL config blocks for each method routed to by `SKILL.md`'s Method Selection table. Base models are never named here — every example uses `BASE_MODEL`/`SFT_CHECKPOINT` placeholders; see `finetuning-method-selection`'s `references/model-catalog.md` for which actual checkpoint to load. All trainer calls use current TRL API conventions (`processing_class`, not `tokenizer=`) — the same conventions established in `lora-qlora-recipes`'s `references/unsloth-trl-mapping.md`. ## DPO — the Default ```python from trl import DPOConfig, DPOTrainer dpo_args = DPOConfig( output_dir="./outputs-dpo", beta=0.1, # settled default learning_rate=7e-7, # 5e-7-1e-6 range — lower than SFT LR num_train_epochs=2, # 1-2 epochs, not more per_device_train_batch_size=4, gradient_accumulation_steps=4, bf16=True, # never fp16 — see lora-qlora-recipes Failure Modes logging_steps=10, seed=3407, ) trainer = DPOTrainer( model=SFT_CHECKPOINT, # policy — starts as a copy of the reference ref_model=None, # None = TRL derives a frozen reference from `model` args=dpo_args, train_dataset=preference_pairs, # {"prompt", "chosen", "rejected"} processing_class=tokenizer, # current TRL — not tokenizer= ) trainer.train() ``` For the iterative on-policy loop described in `SKILL.md`: after each round, load the just-saved checkpoint as both `model` and the frozen reference for the *next* `DPOTrainer` instance — `ref_model=None` on round 1 only; every later round passes the prior round's checkpoint explicitly as `ref_model`. ### Unsloth Wrapper ```python from unsloth import FastLanguageModel, PatchDPOTrainer PatchDPOTrainer() # must run before constructing DPOTrainer model, tokenizer = FastLanguageModel.from_pretrained( model_name=SFT_CHECKPOINT, max_seq_length=2048, load_in_4bit=True, ) model = FastLanguageModel.get_peft_model(model, r=32, lora_alpha=64) # DPOConfig/DPOTrainer usage is unchanged from the plain-TRL block above ``` ## ORPO — Memory-Bound / No SFT Checkpoint ```python from trl.experimental.orpo import ORPOConfig, ORPOTrainer orpo_args = ORPOConfig( output_dir="./outputs-orpo", beta=0.1, # λ in the ORPO odds-ratio term, ≈0.1 learning_rate=2e-5, # 8e-6-5e-5 range num_train_epochs=2, per_device_train_batch_size=4, gradient_accumulation_steps=4, bf16=True, logging_steps=10, seed=3407, ) trainer = ORPOTrainer( model=BASE_MODEL, # no separate SFT checkpoint needed — reference-free args=orpo_args, train_dataset=preference_pairs, # {"prompt", "chosen", "rejected"} processing_class=tokenizer, ) trainer.train() ``` ORPO fuses the SFT and preference objectives into one loss and carries no reference-model memory cost — this is the entire reason it routes in under memory pressure or when no SFT checkpoint exists yet. ## KTO — Unpaired Binary Feedback ```python from trl import KTOConfig, KTOTrainer kto_args = KTOConfig( output_dir="./outputs-kto", beta=0.1, learning_rate=5e-7, # same range as DPO num_train_epochs=1, per_device_train_batch_size=4, gradient_accumulation_steps=4, bf16=True, logging_steps=10, seed=3407, ) trainer = KTOTrainer( model=SFT_CHECKPOINT, ref_model=None, args=kto_args, train_dataset=labeled_examples, # {"prompt", "completion", "label": bool} processing_class=tokenizer, ) trainer.train() ``` `label=True` marks a desirable completion (thumbs-up), `label=False` an undesirable one (thumbs-down) — no pairing between examples is required, and a healthy dataset needs both labels represented, not an all-positive or all-negative set. ## SimPO — Length-Bias Fix, Sweep Required SimPO is reference-free and length-normalized; its published gains are a reported ceiling under a disciplined sweep, not a single-config baseline. Sweep this grid rather than picking one point and trusting it: | Hyperparameter | Sweep range | |---|---| | Effective batch size | 128 (fixed) | | Learning rate | 3e-7 – 1e-6 | | β | 2.0 – 2.5 | | γ/β (target reward margin) | 0 – 1 | ```python from trl.experimental.cpo import CPOConfig, CPOTrainer # TRL implements SimPO via CPOTrainer with loss_type="simpo" simpo_args = CPOConfig( output_dir="./outputs-simpo", loss_type="simpo", beta=2.25, # sweep 2.0-2.5 cpo_alpha=0.0, # 0 disables the CPO NLL term for pure SimPO simpo_gamma=0.5, # gamma/beta sweep point, 0-1 learning_rate=5e-7, # sweep 3e-7-1e-6 num_train_epochs=1, per_device_train_batch_size=4, gradient_accumulation_steps=32, # 4 * 32 = 128 effective batch bf16=True, logging_steps=10, seed=3407, ) trainer = CPOTrainer( model=SFT_CHECKPOINT, args=simpo_args, train_dataset=preference_pairs, # {"prompt", "chosen", "rejected"} processing_class=tokenizer, ) trainer.train() ``` Run this grid as a small sweep (vary `beta`, `learning_rate`, and `simpo_gamma` independently against a held-out preference-accuracy check) before trusting any single point — a SimPO config picked without sweeping is not comparable to the published results this method's gains are cited from. ## Catastrophic Forgetting Across all four methods, a preference-tuned checkpoint that loses general capability is, almost always, a **too-high learning rate** — not an inherent property of the method. Symptoms: fluent output on the preference-tuning task but degraded performance on unrelated held-out capability checks (general QA, format-following the SFT stage previously nailed). Remediation order: 1. Drop the learning rate toward the low end of the method's range in `SKILL.md`'s Method Selection table — this fixes the majority of cases. 2. Reduce epochs (1 instead of 2) if the low-LR run still forgets. 3. Only after 1-2 fail to resolve it, consider a general-data replay mix — mixing 10-30% general instruction data back into the preference run, the same mitigation used against forgetting in `lora-qlora-recipes`-style SFT. A too-low LR under-trains the preference signal instead (the model doesn't change its behavior at all) — if dropping LR removes forgetting *and* removes the intended behavior change, epochs or data quality are the next lever, not pushing LR back up.
-
-
SKILL.md 7.7 KB
--- name: preference-optimization description: Align a fine-tuned model with preference data using DPO, ORPO, KTO, or SimPO. Use when preference pairs or thumbs-up/down feedback exist, when choosing between preference-optimization methods, or when a DPO run needs hyperparameters or debugging. --- # Preference Optimization This skill assumes `finetuning-method-selection` already routed here because the data shape is preference pairs or unpaired thumbs-up/down feedback, not demonstrations (that's `lora-qlora-recipes`) or a verifiable reward signal (that's `grpo-rlvr-training`). What follows is method selection among the DPO family, the evidence for how much that selection actually matters, the production training pattern, and how to build the pairs in the first place. **Input:** a routing decision (preference optimization) plus preference pairs or unpaired feedback, usually from an SFT checkpoint. **Output format:** a validated method choice plus a config — the kwarg values in `references/method-configs.md`, not free-form advice — that `llm-finetuning-training-engineer` consumes directly. ## Method Selection | Data shape | Method | Key parameters | |---|---|---| | Preference pairs, default case | **DPO** | β=0.1, LR 5e-7–1e-6, 1–2 epochs | | Memory-bound or no SFT checkpoint | **ORPO** | reference-free, fused SFT+preference in one loss | | Unpaired thumbs-up/down | **KTO** | binary label per example, no pairing needed | | Length bias observed, sweep budget available | **SimPO** | reference-free; see sweep grid below | - **DPO is the safe default.** Use β=0.1 and a learning rate of 5e-7 to 1e-6 for 1–2 epochs. This LR is *lower* than the SFT LR that produced the checkpoint being aligned — porting an SFT- scale LR into a DPO run is the most common misconfiguration here, not an edge case. - **ORPO** routes in when memory is the constraint, or when there's no separate SFT checkpoint to start from — it's reference-free and fuses the SFT and preference objectives into one loss, skipping the separate SFT pass and the reference-model memory cost DPO carries. - **KTO** routes in when feedback is unpaired binary signal (thumbs-up/down) rather than matched preference pairs — don't force unpaired feedback into synthetic pairs to use DPO instead. - **SimPO** fixes DPO's length bias but only pays off with disciplined sweeping — its published gains are a ceiling reported under a tuned sweep, not a baseline any single config will reproduce. Route here only when there's sweep budget; use DPO instead if there isn't. - **Classic RLHF (reward model + PPO) is retired** outside frontier labs. Don't reach for it in a production pipeline — every method above is cheaper and better-supported for the same data shapes. ### Worked Examples - *"We have an SFT checkpoint and clean paired preference data, no length-bias complaints yet."* → default case → **DPO** at β=0.1. - *"Reviewers click thumbs-up/down per response; nothing is paired."* → unpaired signal → **KTO**, not DPO — don't synthesize pairs to force DPO onto unpaired data. - *"GPU budget doesn't cover a separate SFT pass plus a DPO reference model."* → memory-bound, no separate checkpoint → **ORPO**. - *"DPO output favors longer answers regardless of quality, and there's time to run a sweep."* → length bias plus sweep budget → **SimPO**. Skip it if the sweep budget isn't actually there. ## The Low-Leverage Truth A 2026 240-H100-run study (arXiv 2603.19335) is the load-bearing evidence behind the table above: **loss-function choice is worth roughly 1 percentage point of leverage, model scale is worth roughly 50.** Zero of 20 DPO variants tested beat vanilla DPO. Rankings also **invert with scale** — a variant that wins in a small pilot can lose at deployment size. Two practical consequences: - Don't spend a routing decision agonizing over DPO-variant bake-offs. The table above is sufficient; deeper variant selection is low-leverage compared to data quality and scale. - **Validate at deployment scale before trusting a ranking.** A method comparison run on a small pilot model doesn't transfer to the production size class — re-check the winner once scale changes. This is also why the Method Selection table above is deliberately short: it encodes the ~1pp lever, not a ranking of DPO variants that the same study shows doesn't hold up across scale. Treat any variant-selection advice that isn't in that table — including advice that claims a specific variant "wins" — as unproven until it's been validated at the target deployment size. ## Production Pattern: Iterative On-Policy DPO A single offline DPO pass on a static preference dataset is a starting point, not the production pattern. The policy drifts away from the distribution the pairs were sampled from as training proceeds, and a static dataset goes stale against that drift. Production pipelines run DPO iteratively and on-policy instead: 1. Sample completions from the current policy checkpoint. 2. Score or rank the completions (reward model, judge, or task grader). 3. Run a DPO pass using the current checkpoint as the reference model. 4. The resulting checkpoint becomes both the new policy *and* the new reference for the next round. Repeat. Each round's reference model is the prior round's output, not a fixed initial checkpoint — that's what keeps the preference signal on-policy instead of scoring against an increasingly stale distribution. A single-pass DPO run is still a reasonable first iteration — it just isn't the whole pipeline. Plan for at least one more round once the first checkpoint exists, rather than treating pass one as the finished artifact. ## Pair Construction Build DPO/ORPO pairs from **same-task passing-vs-failing trajectories** — two attempts at the same underlying task, not unrelated best-and-worst examples pulled from different tasks. Within that trajectory set, select the rejected member at **μ−2σ of the reward distribution, never the minimum**. Naive best-vs-worst pair construction (max reward vs. absolute minimum) degrades as scale increases; the μ−2σ selection is more robust to the same scale sensitivity the low-leverage study surfaced above. ``` sorted_by_reward = sort(trajectories, key=reward) chosen = sorted_by_reward[-1] # highest reward mu, sigma = mean(rewards), stdev(rewards) rejected = closest(sorted_by_reward, mu - 2 * sigma) # NOT sorted_by_reward[0] — the absolute minimum # is the naive best-vs-worst construction that # degrades as scale increases. ``` For the mechanics of turning graded traces into these pairs — including rejection sampling and judge-scored delta selection — see `trace-to-training-data`. ## References Complete TRL config blocks per method — `DPOConfig`, `ORPOConfig`, `KTOConfig`, and the SimPO sweep grid — plus Unsloth wrappers and a catastrophic-forgetting note live in `references/method-configs.md`. Those configs use the same current-TRL API conventions established in `lora-qlora-recipes`'s `references/unsloth-trl-mapping.md` (`processing_class`, not `tokenizer=`). `references/method-configs.md` also carries the catastrophic-forgetting note: a too-high learning rate is the usual cause when a preference-tuned checkpoint loses general capability, and the fix is almost always to drop the LR toward the low end of the range in the Method Selection table above before reaching for any other remediation. Related skills: `finetuning-method-selection` routes here once preference pairs or unpaired feedback exist; `lora-qlora-recipes` produces the SFT checkpoint DPO/KTO/SimPO align (ORPO's fused path can skip it); `trace-to-training-data` converts passing/failing trajectories into the pairs this skill's Pair Construction section consumes.
Comments (0)
Sign in to join the conversation.
Reviews (0)
No reviews yet.
No comments yet.