Fine-Tuning LLMs

This post demystifies the internals so the hyperparameters in the code post stop feeling arbitrary. The cover frames fine-tuning as ordinary backpropagation applied gently — the same process that trained the model, just aimed at your data with smaller steps.

If you finish this post understanding why the learning rate is tiny and what catastrophic forgetting is, the later code becomes principled rather than copied.

The core insight is that fine-tuning introduces no new machinery. The forward pass, the loss function, the backward pass, and the optimizer step are identical to pretraining. What changes is only the data and the dials.

This is reassuring and clarifying at once: there's no special 'fine-tuning algorithm' to learn. If you understand how a transformer is trained, you already understand fine-tuning — you just run that same loop on a curated dataset with a much smaller learning rate and far fewer steps.

The cycle diagram lays out the loop you repeat for every batch: a forward pass produces token predictions, the loss measures how wrong they were against the targets, the backward pass computes gradients, and the update step nudges the weights to reduce that loss next time.

Seeing it as a cycle reinforces that training is iterative — thousands of tiny corrections, each one slightly better than the last. No single step matters much; the accumulation is what moves the model.

Understanding where the loss comes from is what separates people who can debug a fine-tune from those who can't. Each training example is a prompt plus the response you want. The model assigns a probability to each possible next token, and cross-entropy loss penalizes it for putting low probability on the correct token.

The crucial detail is masking: you usually compute loss only on the response tokens, not the prompt. You don't want the model learning to generate the instruction — you want it learning to generate the answer given the instruction.

This snippet shows masking concretely. The labels tensor is a copy of the input ids, but the positions corresponding to the prompt are set to -100, the special value the loss function ignores. As a result, cross-entropy is computed only over the response tokens.

Getting this wrong is a common silent bug. If you don't mask the prompt, the model spends capacity learning to reproduce instructions, which both wastes training signal and can degrade the quality of the responses you actually care about.

The learning rate is the single most important dial, and this slide explains the intuition. Pretraining used a comparatively large rate because the model was learning from scratch. Fine-tuning uses a rate one hundred to a thousand times smaller — often between 1e-5 and 2e-4 — because the goal is to nudge already-good weights, not overwrite them.

Too high and the model forgets its general abilities while chasing your examples; too low and it never adapts at all. Most fine-tuning failures trace back to a learning rate that was wrong by an order of magnitude.

Catastrophic forgetting is the failure mode that makes fine-tuning feel risky, so it gets its own slide. When you push too hard on a narrow dataset, the model overwrites the general skills learned in pretraining to fit your handful of examples. It becomes excellent at your task and noticeably worse at everything else.

The defenses are consistent with the learning-rate advice: keep the rate small, run few epochs, and where it matters, mix a slice of diverse general data into your training set so the model has a reason to retain broad ability.

Parameter-efficient fine-tuning is the practical answer to most of the problems above. Instead of updating all the weights, PEFT freezes the original model and trains a small set of new parameters. Because the base weights can't move, catastrophic forgetting is structurally prevented, and you store only the tiny add-on rather than a full model copy.

A further win is modularity: you can train many small adapters for different tasks and swap them on top of the same frozen base. LoRA, covered next, is the most widely used variant of this idea.

The flow diagram shows LoRA's mechanism: the original weight matrix W stays frozen, and a low-rank pair of trainable matrices A and B runs as a side path. The layer's output becomes the frozen W times x plus the small B-times-A-times-x correction.

Because A and B are low rank, they hold a tiny fraction of the parameters of W, yet they're expressive enough to steer behavior. This is why LoRA trains on consumer GPUs and produces adapters measured in megabytes while the base model stays untouched.

This snippet shows how little code LoRA takes in practice. You define a LoraConfig with a rank r, a scaling alpha, the modules to target (commonly the attention query and value projections), and a dropout, then wrap the base model. The print line typically reports something like 0.2 percent of parameters as trainable.

That number is the headline of the whole approach: you're training a fraction of a percent of the model and getting most of the benefit of a full fine-tune, at a fraction of the memory and time.

The dials slide gives sensible defaults so you're not tuning blind. A learning rate around 2e-4 is a good LoRA starting point. One to three epochs is usually enough — more invites overfitting on small sets. A LoRA rank of 8 to 16 is plenty for most tasks; larger rarely helps and costs more. And batch size should be as large as your VRAM allows, with gradient accumulation simulating bigger batches when memory is tight.

These aren't laws, but they're a reliable place to start before you tune based on your own validation curves.

The closing card hands off to post 4, which turns all this theory into a single runnable QLoRA script. You've now seen the loop, the loss, the learning rate, forgetting, and how LoRA sidesteps the worst of it — next you assemble it into working code.

With the mechanics understood, the code post is about wiring, not mystery.