RLHF, DPO, PPO

This post answers the question a decision-maker actually cares about: why does preference tuning matter beyond being a clever research technique. The cover makes the point with history — the gap between GPT-3 and ChatGPT wasn't a leap in scale, it was alignment. The same underlying capability became a usable product once it was tuned to human preferences.

The rest of the post builds that case across three fronts: what alignment buys in usability, what safety and helpfulness it actually encodes, and why the shift from PPO to DPO mattered economically by making alignment affordable.

The first argument is the gap between capability and usability. A pretrained model is essentially a brilliant autocomplete with no manners — it can produce code and poetry yet won't reliably do what you ask, refuses nothing, and treats a harmful prompt identically to a benign one. Raw capability is necessary but nowhere near sufficient for a product.

Preference tuning is the layer that converts that latent capability into reliable, controllable behavior. This reframing matters because people often attribute a chat model's helpfulness to its size or pretraining, when in fact the steerability they're responding to comes almost entirely from the alignment step layered on top.

This slide gives the standard vocabulary for what alignment optimizes: helpful, honest, and harmless. Helpful means actually answering the question rather than dodging or rambling. Honest means not fabricating with confidence. Harmless means refusing or redirecting genuinely dangerous requests. These three goals frequently trade off against one another.

Preference data is how those trade-offs get taught. When annotators prefer one answer over another, they're implicitly weighing helpfulness against safety against honesty, and the model learns to balance them the way the labelers did. Framing alignment as the three H's gives readers a compact way to reason about what a given preference dataset is really optimizing for.

The bar chart visualizes the value alignment adds at each stage. The base language model scores low as an assistant — fluent but unsteerable. Supervised fine-tuning lifts it substantially by teaching instruction-following. RLHF or DPO on top pushes it higher still by making it reliably helpful and safe.

The numbers are illustrative, but the shape is the lesson: most of the 'assistant' quality users perceive comes from the SFT and preference-tuning stages, not from pretraining alone. It's a concrete way to show that alignment isn't a cosmetic finish — it's responsible for a large fraction of what makes a model feel usable.

This slide explains a foundational design choice in preference learning: humans compare rather than grade. Asking an annotator to assign an absolute score to an answer produces noisy, inconsistent numbers that drift between people and even within the same person over time. Asking which of two answers is better produces a clean, reliable signal.

This is why every method in the day is built on pairwise preference data. Relative comparisons are cheaper to collect and far more consistent across annotators, which means the reward model or DPO loss is learning from a higher-quality signal. The choice isn't an implementation detail — it's the reason preference tuning works as reliably as it does.

Here we introduce the central risk of preference tuning: the reward is only a proxy. A reward model approximates human taste, but it's imperfect, and any imperfect objective can be exploited. Push the policy hard enough to maximize reward and it will discover inputs that score high while actually being worse — needlessly long answers, sycophantic agreement, evasive hedging.

This is reward hacking, and it's the conceptual root of many alignment failures. The slide sets up why the KL penalty isn't optional: it's the main defense against the policy wandering off to exploit the proxy. Understanding that the reward and the true goal are different things is the single most important idea for reasoning about what can go wrong.

The flow diagram makes reward hacking concrete in four steps. The policy optimizes to push its reward score up. In doing so it finds a loophole — say, becoming longer and more agreeable. The reward model, which weakly correlates length and agreement with quality, assigns a high score, so from the optimizer's view it's winning. But a human looking at the output judges it worse.

Drawing the loop this way shows that reward hacking isn't a bug in the code; it's the predictable result of optimizing a proxy too hard. The gap between 'score: high' and 'human: bad' is exactly what the KL leash and good preference data are meant to keep small.

This tips slide enumerates what the KL leash actually buys, so it's seen as a feature rather than a mysterious penalty term. It keeps the model's language fluent and on the distribution it was pretrained on, preventing the degenerate text that pure reward-chasing produces. It blocks the worst reward exploits by penalizing large departures from the reference. In effect it trades a little raw reward for a lot of behavioral sanity.

The final bullet names the control knob: beta (in DPO) or the KL coefficient (in PPO) sets how tight the leash is. This previews post 3's mechanics and post 5's warning that beta is one of the most consequential and most commonly mis-set hyperparameters in the whole pipeline.

This slide makes the economic argument that explains why DPO was such a big deal. Classic PPO requires roughly four models resident in memory — policy, reference, reward model, and a value critic — plus an online sampling loop and a set of notoriously touchy hyperparameters. That combination demands serious infrastructure and RL expertise.

DPO collapses all of that into a single supervised-style loss computed over preference pairs, with no reward model and no rollouts. The practical effect is that alignment dropped from a frontier-lab undertaking to something that fits the hardware and skill budget of a normal fine-tuning team. That accessibility, not a quality leap, is the heart of why DPO spread so quickly.

This comparison lays the cost structures side by side for planning. PPO entails standing up a reward model and RL infrastructure, sampling from the policy online at every step, juggling many fragile knobs, and committing lab-scale effort. DPO needs only preference pairs, runs offline with no sampling, exposes essentially one main knob (beta), and costs about as much as an ordinary fine-tune.

The right-hand column is effectively a list of barriers removed. The takeaway for a practitioner is that unless you specifically need online exploration or a live, evolving reward signal, DPO gets you the same alignment objective at a fraction of the operational complexity — which is why it's the default starting point for most teams now.

This snippet turns the capability-versus-alignment claim into illustrative numbers. Using a human-judged win-rate against a strong reference, a raw base language model wins only a small fraction of comparisons because it ignores intent. The SFT model improves markedly by following instructions. The DPO-tuned model improves again by matching preferences.

The punchline is the print statement: a large swing in win-rate from preference tuning alone, with the same weights underneath. The point isn't the exact figures but the message that the delta comes from alignment, not from adding parameters or training data scale. Running an analogous evaluation on your own task is the honest way to confirm that preference tuning actually earned its keep.

Closing the 'why' post with a caution keeps the case credible. Preference tuning improves average behavior; it does not make a model provably safe. Jailbreaks find prompts the training never covered, distribution shift exposes gaps, and biased preference data quietly bakes in the labelers' blind spots. None of these are eliminated by a clean RLHF or DPO run.

The practical framing is that alignment raises the floor on behavior rather than issuing a safety certificate. Teams that internalize this keep investing in evaluations, red-teaming, and runtime guardrails instead of treating a tuned model as finished. This sets up post 5, where several of these failure modes return as concrete, common mistakes.

The closing card hands off to post 3, the mechanics, which opens the hood on the reward model's loss, PPO's four models and clipped update, and the derivation that lets DPO drop the reward model entirely. This post argued why alignment matters and why DPO made it affordable; the next shows precisely how both methods achieve the same objective.

The handoff is intentional: once you believe alignment is worth doing and now feasible, the natural next question is how the optimization actually works — and that's exactly post 3.