RLHF, DPO, PPO

This is the hands-on post, so the cover makes a concrete promise: a complete DPO fine-tune you can actually run, with no reward model and no RL loop. The aim is that a reader can load an SFT model, feed it chosen/rejected pairs, train, and get aligned outputs the same session.

Everything uses TRL's DPOTrainer because that's the standard, well-supported path, and the post also shows the skeleton of a PPO loop near the end so the reader can see directly why DPO is the simpler route most teams now choose.

The overview slide gives the four stages so each code slide has a clear home: load an SFT model and its reference, format chosen/rejected pairs, train with DPOTrainer and a beta value, then generate from the aligned model. Naming the sequence up front turns the code that follows into a map a reader can navigate rather than a wall of snippets.

This ordering deliberately mirrors the pipeline diagram later in the post, so the same workflow is reinforced once as a checklist and once visually.

Step one loads the starting model, and the comment carries the most important lesson: DPO begins from an SFT model, not a raw base. The chosen checkpoint is an already-instruction-tuned model, loaded in bfloat16 for efficiency. The pad token is set to the eos token because causal models often lack a dedicated pad token, and omitting this causes confusing batching errors.

The specific model is incidental — any competent SFT checkpoint works and the rest of the script is unchanged. The emphasis on starting from SFT is intentional foreshadowing: post 5 lists 'starting from a base model' as a common and costly mistake, because preference tuning has nothing to refine without prior instruction-following behavior.

Step two formats the preference dataset into the exact three columns DPOTrainer expects: prompt, chosen, and rejected. The example uses Anthropic's hh-rlhf dataset, which stores full conversation transcripts, so the to_pref function splits off the shared prompt and isolates each branch's final answer as the chosen or rejected completion.

The details here matter more than they look. The prompt must be identical across the chosen and rejected examples — the only difference is the answer — because DPO compares the model's log-probabilities of two completions to the same context. Getting the column names or the prompt/completion split wrong is one of the most common reasons a DPO run silently learns nothing useful.

Step three configures and runs the training, and the single most important argument is beta. The DPOConfig sets one epoch, a small per-device batch of two with gradient accumulation of eight, a deliberately low learning rate of 5e-6, bf16, and beta at 0.1. The comment flags beta as the KL strength — the leash from the mechanics post — controlling how far the policy may drift from the reference.

The DPOTrainer ties the model, arguments, dataset, and tokenizer together; if no separate reference model is passed it automatically creates a frozen copy of the start model to serve as the KL anchor. The low learning rate is not arbitrary — DPO is sensitive, and a rate that would be fine for ordinary fine-tuning can push the policy into degenerate text here.

The pipeline diagram restates the entire workflow visually so the code slides cohere into one picture: load the SFT model as both policy and reference, format the prompt/chosen/rejected pairs, run DPO training with beta as the leash, then generate aligned answers. Placing it right after the training code helps a reader zoom back out from line-level detail to the overall shape before moving to inference.

This is the same four-stage flow named in the overview slide, now drawn — the repetition is intentional reinforcement, not filler.

Step four generates from the aligned model the way you'd actually use it. The saved DPO checkpoint is reloaded, a prompt is formatted with the same Human/Assistant structure the training data used, and generation runs greedily with do_sample set to False for reproducible output. Decoding with skip_special_tokens cleans up the result.

The formatting discipline is the subtle point: the prompt template at inference must match what the model saw during training, or quality silently degrades. This step closes the loop the whole post built toward — the model you just tuned is now producing an aligned answer to a real prompt, with the preference signal from the dataset visible in how it responds.

The 'lines that matter' slide extracts the four details that most affect results so they don't get buried in the code. Start from an SFT model, not a base, or there's nothing to refine. The dataset columns must be exactly prompt, chosen, and rejected. Beta is the KL leash and 0.1 is a sensible starting point. And the learning rate should be low — around 5e-6 — because DPO is sensitive and a too-high rate causes drift.

These four are precisely the points that separate a script that merely runs from one that produces a genuinely aligned model. They map directly onto the most common mistakes catalogued in post 5, which is why they're worth memorizing as a short pre-flight list.

This slide shows the bones of a PPO loop purely for contrast, so the reader can see what DPO replaces. The PPOTrainer is set up with a policy, a reference model, and a tokenizer, and the loop does three explicit things per batch: generate responses from the current policy, score them with a reward model, and apply a clipped update via ppo.step.

The comment makes the comparison explicit — those three stages plus a separately trained reward model are exactly what DPO's single loss collapses into one supervised-style step. Seeing the loop laid out, with its sampling and external scoring, makes the simplification concrete: DPO isn't magic, it's the closed-form shortcut through this loop derived in post 3.

This mistake slide isolates a DPO-specific bug: the reference model. DPO computes log-ratios between the policy and a frozen reference, so the reference is what defines the KL leash. If you forget it or point it at the wrong checkpoint, the KL term becomes meaningless and the policy can drift into gibberish that the loss still scores well — a silent failure with a falling loss curve.

The fix is to ensure the reference is your real SFT model, frozen. TRL creates a copy of your start model by default, which is correct as long as that start model is the right SFT checkpoint. Calling this out as its own slide, right after the code, is meant to burn the rule in before a reader hits it in a real run.

This snippet sketches the sanity gate every alignment fine-tune should pass before you trust it. It runs the tuned model on a held-out prompt, asserts that the output is actually different from the prompt (a cheap guard against a model that produces nothing or echoes its input), and prints the result for a human to eyeball: is it helpful, on-topic, and does it stop cleanly?

This is deliberately minimal — the real evaluation, shown in post 5, compares win-rates against the reference on a held-out set. But even this lightweight check catches the most embarrassing failures early. The principle is that a falling DPO loss is necessary but never sufficient evidence of success; you must look at generations.

The closing card hands off to post 5, the common mistakes, which catalogs the quiet failure modes of preference tuning — noisy data, a too-loose beta, the wrong reference, rewarding length and sycophancy, and trusting the loss curve. You now have a working DPO script; the final post is about ensuring what it produced is actually better.

The pairing is intentional: runnable code here, and the judgment to validate its output next.