RLHF, DPO, PPO

This closing post is about the alignment failures that don't announce themselves with an error. A run can train cleanly, show a falling DPO loss or rising PPO reward, and finish without a crash — yet produce a model that's become a confident sycophant, agreeing with everything and answering nothing. The cover names that uncomfortable reality directly.

The goal is to inoculate readers against the specific, common traps so they stop paying for them in wasted GPU hours and shipped regressions. Each mistake here is one a careful practitioner is likely to hit, and each comes with the habit that catches it.

The first trap is noisy preference data, and it's the most fundamental because preference tuning can only ever be as good as its comparisons. Inconsistent annotators, ambiguous prompts, or a labeling guideline that quietly rewards style over substance teach the model exactly the wrong thing — and it learns it confidently, because the loss has no way to know the labels were bad.

No loss function rescues contradictory labels; if two annotators disagree on similar pairs, the model just averages the confusion. The real lever is upstream: clean, agreed-upon comparisons with clear guidelines. This is why mature alignment efforts spend heavily on annotation quality and inter-annotator agreement before they ever touch a hyperparameter.

The second trap is setting beta — the KL strength — too low. Beta is the leash, and a loose leash lets the policy drift far from the reference in pursuit of reward. In DPO that drift shows up as degenerate text: repetition, hedging, or strange phrasing the loss happens to favor. In PPO it manifests as outright reward hacking against the reward model. Set beta too high, on the other hand, and the model barely changes from the reference.

The guidance is to start near 0.1 and tune from there, and crucially to watch the KL divergence, not just the loss. A falling loss with a runaway KL is the signature of a model gaming its objective. Beta is the single most consequential dial in DPO, and treating it as fixed is how runs quietly go wrong.

The bar chart makes the beta trade-off concrete. A very low beta of 0.01 scores poorly because the policy drifts and hacks the reward signal. A beta of 0.1 is the balanced default that lets the model improve while staying on-distribution. A high beta of 0.5 also scores lower because the leash is so tight the model barely moves from the reference.

The numbers are illustrative, but the inverted-U shape is the real lesson: more freedom is not monotonically better, and neither is more constraint. The best beta is the one that allows the behavior change you want while keeping generations fluent and honest — which is why it has to be tuned against a held-out eval rather than guessed once.

The third trap is starting preference tuning from a raw base model. Both DPO and PPO assume a competent starting point — a model that already follows instructions. Run them on a pretrained base and there's no good behavior to refine; preference tuning just amplifies whatever noise the base produces, often making things worse.

The fix is to always do supervised fine-tuning first, and to ensure the reference model used for the KL term is that real SFT checkpoint, not the base. This connects directly to the mechanics: the KL leash measures distance from the reference, so if the reference is the wrong model, the leash is anchored to the wrong place and the whole constraint is meaningless. SFT-first is not optional — it's a precondition for preference tuning to work at all.

This code slide operationalizes the reference-model check into a few assertions. The reference must be in eval mode (not training) and every one of its parameters must have requires_grad set to False, confirming it's truly frozen. The comment adds the second, easier-to-miss requirement: the reference must be the same checkpoint the policy was initialized from, or the KL term measures distance from the wrong starting point.

Wiring these assertions into your setup is cheap insurance against two silent failures — accidentally training the reference, and anchoring the leash to the wrong model. Both produce runs that look fine in the logs and ship a worse model, which is exactly the kind of quiet failure this post exists to prevent.

The fourth trap is rewarding superficial correlates of quality instead of quality itself. Reward models — and by extension the preferences they're trained on — tend to latch onto easy proxies: longer answers, more hedging, more flattery. Optimize against those and you get verbose, sycophantic responses that score high on the proxy and help the user less.

The defenses are concrete: length-normalize so the model isn't rewarded simply for writing more, and deliberately include preference pairs where the shorter or more direct answer is the chosen one. This teaches the reward signal that conciseness and directness are valued. Sycophancy and verbosity are not random failures — they're the predictable result of a reward signal that conflates length and agreeableness with helpfulness.

The fifth trap is trusting the loss curve, and it's the most seductive because the numbers look like success. A falling DPO loss or a rising PPO reward only means the model is getting better at the game you defined — maximizing the proxy. It says nothing directly about whether the model is actually more helpful, honest, or safe, because the reward is a proxy for those things, not the things themselves.

The only honest check is to look at generations on held-out prompts the training never saw, judged either by humans or by a strong model against the reference. Keeping a frozen eval set and measuring win-rate on it is the discipline that separates real improvement from a model that merely learned to win its own scoring function.

The decision diagram turns 'which method should I use' into a quick flowchart. If you have clean preference pairs and limited infrastructure, use DPO — it's simpler and more stable and targets the same objective. If you instead need online exploration or a live, evolving reward signal that static pairs can't provide, PPO and full RLHF are warranted despite the extra machinery. And if your preference data is noisy or thin, the right move is neither method yet — collect better pairs first.

Framing it as a decision tree keeps practitioners from reaching for PPO's complexity by default or, worse, running any preference method on bad data. The honest answer for most teams most of the time is the DPO branch, with PPO reserved for the cases that genuinely need online RL.

This snippet sketches the alignment sanity gate that should run before anything ships. It judges the tuned model against the reference on a set of held-out prompts — using either human raters or a strong LLM as the judge — counts how often the tuned model wins, and asserts that the win-rate clears 0.5, i.e. that the tuning actually helped rather than hurt. The comment adds the essential safety step: red-team to confirm that safety behavior survived the tuning.

The judging function is task-specific, but the gate pattern is universal: never ship a preference-tuned model on the strength of its loss curve alone. Making 'beat the reference on a held-out eval' an enforced assertion, plus an explicit safety check, is what keeps regressions and safety erosion from reaching users.

The pre-flight checklist consolidates the post into five habits: audit the preference data for noise and bias, start from an SFT model and freeze the correct reference, tune beta while monitoring KL rather than just the loss, length-normalize and reward substance over verbosity, and beat the reference on a held-out eval before shipping. Run through it before and after every preference-tuning run.

Notice that these are mostly about data quality and measurement, not modeling cleverness. That's the real lesson of the day: getting good results from RLHF, DPO, or PPO is less about exotic settings and more about clean comparisons, sane defaults, the right reference, and proving the result is genuinely better and still safe.

The final slide is the day's closing caution, and it deliberately echoes the warning from post 2: do not mistake aligned for safe. Even a clean run only raises the floor on behavior. Jailbreaks find uncovered prompts, distribution shift exposes edge cases, biased pairs leak the labelers' blind spots, and over-optimization can erode the very safety behavior you trained in.

Alignment is one layer in a defense-in-depth strategy, not a certificate. Pairing it with red-teaming, evaluation suites, and runtime guardrails is what makes a tuned model trustworthy in production. With the concept, the economics, the mechanics, a full runnable DPO script, and the common pitfalls all covered, you can now run RLHF, DPO, and PPO end to end — and, just as importantly, judge honestly whether your result earned its place in front of users.

The closing card wraps the entire day. With the concept, the economics, the mechanics, a full runnable DPO script, and the common pitfalls covered, you can now run RLHF, DPO, and PPO end to end — train on preference pairs, set beta sensibly, and validate the result against a held-out eval.

That last part is the real skill. The most valuable thing here isn't launching a trainer; it's the judgment to choose DPO or PPO for the job, anchor the right reference, and prove — against a baseline, with safety checks — that your aligned model genuinely earned its place in production.