RLHF, DPO, PPO

This post turns RLHF, PPO, and DPO from concepts into concrete mechanics, so the hyperparameters in the code post stop feeling like incantations. The cover frames the key realization: PPO and DPO optimize the same objective, one through a noisy RL loop and the other through a clean closed-form loss.

If you finish this post understanding the Bradley-Terry reward loss, the four models PPO juggles, the clipped objective and KL term, and how DPO folds the reward into a single loss, then the code in post 4 becomes wiring you understand rather than settings you copy.

This slide explains how the reward model is trained. Given a prompt with a chosen and a rejected answer, the reward model r(x,y) must assign the chosen answer a higher score than the rejected one. The Bradley-Terry model turns that requirement into a probability — the chance that chosen beats rejected is the sigmoid of their score difference — and the loss maximizes the log of that probability.

Concretely the loss is the negative log-sigmoid of (r_chosen minus r_rejected), averaged over the dataset. Minimizing it widens the margin between good and bad answers in the reward model's eyes. This same Bradley-Terry formulation reappears inside DPO's loss later in the post, which is no coincidence — it's the shared statistical model of preferences underneath both methods.

This code slide shows the reward loss in a few lines so the math becomes tangible. The function takes the scalar rewards for the chosen and rejected responses in a pair and returns the negative log-sigmoid of their difference, averaged over the batch. F.logsigmoid is used directly for numerical stability rather than composing log and sigmoid by hand.

The comment captures the intuition: a lower loss corresponds to a bigger margin between the model's score for good answers versus bad ones. Seeing the reward objective as a handful of lines demystifies it — it's just a pairwise ranking loss, the same kind used in many recommendation and ranking systems, applied to model outputs.

The stack diagram inventories the four models PPO keeps in play, which is the single biggest source of its operational weight. The policy is the model actively being trained. The reference is a frozen copy of the SFT model that anchors the KL penalty. The reward model, also frozen, scores generated responses. The value or critic network estimates the expected reward, which is needed to compute advantages.

Laying them out as a stack makes the memory and complexity cost vivid: four sets of weights, three of them queried every step. This is precisely the burden DPO removes — it needs only the policy and a frozen reference, dropping the reward model and critic entirely. Seeing the four here is what makes DPO's simplification feel significant rather than abstract.

This slide walks through the PPO loop conceptually. First the current policy samples responses to a batch of prompts. Those responses are scored by the reward model. The critic predicts how much reward to expect, and the difference between actual and expected reward is the advantage — a measure of how much better a response was than the baseline. Finally the policy is updated to make high-advantage responses more likely.

The crucial qualifier is the clip: the update is capped so no single step moves the policy too far. RL on language models is unstable, and without that cap a single batch can shove the policy into a region where the reward model's scores are no longer trustworthy. The loop is powerful precisely because it explores, but that exploration is what demands the guardrails described next.

This code slide shows PPO's clipped surrogate objective, the heart of the algorithm. The ratio is how much more or less likely the new policy is to take an action versus the old policy. The objective multiplies that ratio by the advantage, but also computes a version where the ratio is clamped to a narrow band around one, and takes the minimum of the two. That minimum is what prevents an over-large update.

The comment underneath is just as important: the reward fed into PPO isn't the raw reward-model score. It's that score minus a per-token KL penalty against the reference policy, scaled by beta. So the KL leash is built directly into the reward signal, not bolted on afterward. Together the clip and the KL-shaped reward are the two mechanisms that keep PPO from flying apart.

This slide explains why the clip and the KL term both exist, since they solve related but distinct problems. The clip is about optimization stability: it stops a single batch from yanking the policy into a region where the reward model has never seen examples and its scores are unreliable, which is how RL runs diverge. The KL penalty is about staying on-distribution: it keeps generations fluent and close to the reference so the model doesn't drift into degenerate text that happens to score well.

Together they're the guardrails that make reward-chasing safe. Remove the clip and training becomes unstable; remove the KL and the policy exploits the reward model. Understanding that these are two separate safeguards — one for the optimizer, one for the output distribution — is key to debugging PPO when it misbehaves.

This slide delivers DPO's central insight, which is genuinely elegant. DPO starts from the exact same KL-constrained objective PPO optimizes — maximize reward while staying close to the reference — and instead of solving it by iterative sampling, solves it analytically. The math shows that the optimal policy under that objective has an implicit reward equal to beta times the log-ratio of the tuned policy to the reference.

The consequence is profound: if the reward is just a function of the policy's own log-probabilities, you don't need a separate reward model at all. You can substitute that implicit reward back into the Bradley-Terry preference loss and optimize the policy directly on chosen/rejected pairs. The reward model, the sampling loop, and the critic all vanish — replaced by a single closed-form loss.

This comparison diagram puts the two routes side by side now that both are explained. The PPO route fits a reward model, samples from the policy, computes scores and advantages, and applies a clipped RL update — an iterative, online procedure. The DPO route skips the reward model, uses the chosen and rejected answers directly, treats the policy's log-ratio as the implicit reward, and takes a single gradient step on a supervised-style loss.

Seeing them aligned this way reinforces the day's thesis: same destination, very different path. PPO explores its way to the optimum through trial and error; DPO jumps straight there using the closed-form solution. The diagram is the visual bridge between the conceptual comparison in post 1 and the runnable code in post 4.

This code slide presents the full DPO loss, which is remarkably compact. For each pair it computes the chosen log-ratio — the policy's log-probability of the chosen answer minus the reference's — and the rejected log-ratio likewise. The loss is the negative log-sigmoid of beta times the difference between those two log-ratios, averaged over the batch.

Notice the structure: it's the same Bradley-Terry log-sigmoid form as the reward loss from earlier, but with the implicit reward (the log-ratio) standing in for an explicit reward-model score. That's the derivation made concrete. The reference log-probabilities are computed once with a frozen model, so in practice DPO is barely more expensive than ordinary supervised fine-tuning — which is exactly why it scaled the way it did.

This slide explains beta's role in DPO so the reader can tune it deliberately. Beta is the KL strength, but unlike PPO where the KL is a separate reward term, here it's baked directly into the loss as the multiplier on the log-ratio difference. A small beta lets the policy move far from the reference, enabling larger behavior shifts at the risk of drift. A large beta keeps the policy conservative and close to the reference.

It plays exactly the same role as PPO's KL coefficient — the leash — but as a single, stable hyperparameter rather than one knob among many fragile ones. This is why post 5 singles out beta as the most consequential dial in DPO: set it too low and the model drifts or reward-hacks, too high and it barely changes.

This closing-mechanics slide drives home the relationship between the two methods, correcting the common misconception that they're unrelated algorithms. DPO is not a different objective from PPO — it's a closed-form solution to PPO's exact objective under a Bradley-Terry preference model. PPO reaches that objective the hard way, through iterative sampling and clipped updates; DPO derives the answer and optimizes it directly.

Framing them as the same destination via different paths is the conceptual payoff of the whole post. It explains why DPO can be so much simpler yet target the same alignment goal, and it equips the reader to reason about when the extra machinery of PPO is actually worth it — namely when you need the online exploration that a static set of preference pairs can't provide.

The closing card hands off to post 4, which assembles these mechanics into one runnable DPO fine-tune. You've now seen the reward loss, the four-model PPO loop, the clipped objective, the KL term, and the DPO derivation — the next post is about wiring the DPO version together correctly with real tooling.

With the objective understood, the code becomes a matter of arranging known pieces rather than trusting unfamiliar settings.