RLHF, DPO, PPO

This post opens the day by separating two things people constantly conflate: a model that predicts text well and a model that behaves like a helpful assistant. The cover frames RLHF, DPO, and PPO as the family of techniques that bridge that gap by training on human preferences rather than raw text imitation.

The rest of the post installs the core mental model — supervised fine-tuning, then a reward signal from human comparisons, then optimization against it — so that the mechanics and code in later posts feel like consequences of one idea rather than a pile of unrelated tricks.

The definition deliberately keeps three terms distinct. RLHF is the overall idea: tune a language model to match human preferences. PPO is the specific reinforcement-learning algorithm that the original RLHF papers used to do the tuning. DPO is a later method that hits the same target with a single supervised-style loss and no RL loop at all.

Keeping these separate matters because beginners often treat them as competing algorithms when really RLHF is the goal and PPO and DPO are two routes to it. Throughout the day, 'classic RLHF' means the PPO pipeline, and DPO is the simpler modern alternative that optimizes the same underlying objective.

This slide makes the case for why plain language modeling isn't enough. Pretraining optimizes a single objective — predict the next token — over a vast scrape of the internet. That teaches the model what text looks like, including all the unhelpful, unsafe, and rambling text it ever saw. There is simply no term in that loss for 'this response is more helpful than that one.'

The consequence is a model that is fluent but unsteerable: it will continue a toxic prompt as readily as a benign one and has no built-in notion of following your actual intent. Preference tuning adds exactly the missing signal — a way to say one response is better than another — which is the entire reason the rest of the pipeline exists.

Here we pin down the raw material of all three methods: preference data. Each example is a prompt plus two candidate answers, with a human label marking one as chosen and the other as rejected. Crucially, you don't need a gold-standard perfect answer — you only need a relative judgment about which of two responses is better.

This matters because relative comparisons are dramatically easier and more consistent for humans to produce than absolute scores. Asking someone to rate an answer 73 out of 100 yields noise; asking which of two answers they prefer yields a clean, reliable bit of signal. That reliability is what makes preference learning work at scale.

This code slide makes preference data tangible by showing a single example as a plain dictionary: a prompt, a chosen answer that's genuinely helpful, and a rejected answer that dodges the question. There's no numeric score anywhere — the entire signal is the relative judgment that the chosen answer beats the rejected one.

The comment underneath is the key point: thousands of pairs like this *are* the training data. Whether you go the PPO route (training a reward model on these pairs) or the DPO route (optimizing on them directly), this dictionary is the atom everything is built from. Seeing it as ordinary code, rather than an abstract 'preference dataset,' makes the rest of the day concrete.

The pipeline diagram lays out classic RLHF as three sequential stages so the reader has a scaffold for everything that follows. First, supervised fine-tuning (SFT) teaches the base model to follow instructions at all. Second, a reward model learns human taste from the preference pairs. Third, an RL algorithm — PPO — optimizes the policy to score well under that reward model.

Showing it as a pipeline clarifies that these stages build on each other: each one assumes the previous succeeded. DPO, introduced later in the day, essentially fuses the second and third stages into a single loss, but the SFT stage remains the shared starting point for both approaches.

This slide explains stage one, supervised fine-tuning, which is the foundation the entire day rests on. Before any preferences enter the picture, you fine-tune the base model on curated prompt-and-answer demonstrations so it learns the basic shape of following instructions. The result is the SFT model.

Both PPO and DPO begin from this SFT checkpoint, not from the raw pretrained base. The reason is practical: preference tuning refines existing good behavior, and if there's no instruction-following behavior to refine, it has nothing reasonable to work with. This point returns as a common mistake in post 5, where starting from a base model is flagged as a frequent and costly error.

Stage two is the reward model, and this slide explains its role as a learned stand-in for human judgment. Trained on the preference pairs, the reward model takes a prompt and a response and outputs a single scalar predicting how much a human would like that response. It compresses thousands of human comparisons into a function you can query cheaply.

That cheap querying is the whole point: the RL stage needs to score enormous numbers of generated responses, far more than humans could ever label directly. The reward model becomes the proxy judge that makes online RL feasible. Its imperfection as a proxy, however, is exactly what creates the reward-hacking risk explored in post 2.

This comparison gives the reader the punchline of the day early: PPO and DPO are two ways to reach the same goal. The PPO column lists the classic RLHF machinery — a separately trained reward model, an RL loop that generates and scores responses, and as many as four models held in memory at once. It's powerful but operationally fiddly.

The DPO column shows the simplification: no separate reward model, no sampling loop, just a single loss computed over the chosen and rejected pairs. It's simpler and notably more stable to train. Seeing them side by side this early sets up the deeper mechanics in post 3, where DPO is shown to be a closed-form solution to PPO's exact objective.

The KL-leash slide introduces a concept that recurs throughout the day, so it earns a definition of its own. Both PPO and DPO pull the model toward preferred answers, but unconstrained that pull would let the model contort itself into whatever degenerate text scores well. The KL-divergence penalty against the original SFT model acts as a leash, keeping the tuned model close to a sane, fluent baseline.

The framing 'chase reward, but stay close to a sane base' captures the central tension of all preference tuning. In PPO this leash is a KL term added to the reward; in DPO it's the beta coefficient baked into the loss. Either way, alignment is always a balance between optimizing the preference signal and not drifting into nonsense.

This tips slide grounds the abstractions in why anyone should care, one line at a time. Preference tuning is what turns 'sounds fluent' into 'is actually helpful.' It's the mechanism by which fuzzy human values get encoded as a concrete training loss. It's the technique behind every major chat assistant — ChatGPT, Claude, Llama-Chat. And the arrival of DPO is what made running this kind of alignment cheap enough for ordinary teams.

These four points preview the 'why it matters' arc of post 2 without diving into the economics yet. The goal here is simply to convince the reader that the concept is worth the effort of understanding the mechanics that follow.

Closing the concept post with explicit limits prevents the most common overclaims. Preference tuning reshapes behavior and tone using capabilities the model already has; it does not inject new factual knowledge, so it's no substitute for retrieval or further pretraining. It also cannot rescue bad preference data — feed it contradictory or biased comparisons and you get a model that's confidently wrong in human-approved ways.

The deepest caveat is that the reward is always a proxy for what you actually want, never the thing itself. This single idea underlies reward hacking, sycophancy, and most of the failure modes in posts 2 and 5. Naming it now keeps the reader from treating a high reward or a low loss as proof of a good model.

The closing card points to post 2, which makes the case for why this matters — the difference between a clever model and a usable one. This post established the map: SFT, reward, optimization, and the two routes through it. The next translates that map into the practical value alignment delivers and the economics that DPO changed.

The day's arc is the familiar one: concept, then why it matters, then mechanics, then code, then pitfalls.