Fine-Tuning LLMs

This is the hands-on post, so the cover promises something concrete: an end-to-end LoRA fine-tune that runs on a single GPU. The goal is that a reader can copy these snippets, point them at their own data, and have a tuned model the same day.

Everything here uses the QLoRA recipe — 4-bit base model plus LoRA adapters — because that's the workflow that actually fits on accessible hardware and dominates real-world practice.

The overview slide sets expectations for the five concrete stages: load a base model in 4-bit to save memory, format an instruction dataset, attach LoRA adapters, train and save, then load and run inference. Naming the stages up front gives readers a map so each code slide has a clear place.

This structure mirrors the pipeline diagram later in the post, reinforcing the same sequence twice — once as a checklist, once as a visual.

Step one loads the model in 4-bit using bitsandbytes, which is what makes QLoRA fit on modest GPUs. The BitsAndBytesConfig requests nf4 quantization with a bfloat16 compute dtype — a combination that preserves quality while roughly halving memory versus 16-bit.

The last line, setting the pad token to the eos token, is a small but important detail for causal models that lack a dedicated pad token. Skipping it causes confusing errors during batching, so it belongs in every script of this kind.

Step two is data formatting, and it's where most quality is won or lost. The to_text function wraps each example in a fixed instruction/response template and — critically — appends the eos token after the response so the model learns where to stop.

The template is arbitrary, but consistency is not: every row must use the exact same structure, and the same template must be reused at inference. The dataset is loaded from a JSONL file, which is the standard format for instruction data because each line is one independent example.

Step three defines both the LoRA configuration and the training arguments. The LoraConfig sets rank 16, alpha 32, a small dropout, and targets the query and value projection modules — sane defaults that work across many models. The task type marks this as causal language modeling.

The SFTConfig sets the training schedule: two epochs, a per-device batch of four with gradient accumulation of four (an effective batch of sixteen), a 2e-4 learning rate, and bf16 precision. These mirror the defaults justified in post 3, so nothing here should feel arbitrary.

Step four is the actual training. The SFTTrainer from TRL ties together the model, dataset, LoRA config, and arguments, handling tokenization and the training loop for you. Calling train runs the fine-tune; save_model writes only the LoRA adapter weights, which are typically just megabytes.

That last point is a practical highlight: because you save adapters rather than the full base model, fine-tunes are cheap to store and trivial to version. You can keep dozens of task-specific adapters without keeping dozens of multi-gigabyte models.

The pipeline diagram restates the whole workflow visually so the five code slides cohere into one mental picture: load the 4-bit base, format the instruction pairs, attach LoRA adapters, train with the SFTTrainer, then load and infer.

Having the visual immediately after the training code helps readers zoom back out from the line-level detail to the shape of the process before moving on to inference.

Step five shows inference, and it deliberately reloads the base model and attaches the saved adapter with PeftModel, mirroring how you'd serve in production rather than reusing the in-memory training object. The prompt uses the exact same template as training — same headers, same spacing — which is essential for good output.

This closes the loop: the model you trained is now answering a real prompt, and the formatting discipline introduced in step two pays off here directly.

The 'lines that matter' slide pulls out the four details that most affect results so they don't get lost in the code. nf4 plus bf16 is what fits the model in roughly half the VRAM. The eos token after each response is what teaches the model to stop. Consistent formatting across every row is what makes training signal clean. And save_model storing adapters rather than the base is what keeps fine-tunes lightweight.

These are exactly the points that separate a script that works from one that silently underperforms.

The mistakes slide previews post 5 from a code angle, naming the failures that hide in working scripts. Forgetting the eos token produces a model that rambles past where it should stop. A formatting mismatch between training and inference quietly tanks quality because the model sees out-of-distribution input. And too many epochs on a small set overfits fast.

Each of these compiles and runs without error, which is exactly why they're dangerous — the only defense is knowing they exist and watching the loss curve.

The optional merge step shows how to fold the adapter back into the base weights with merge_and_unload, producing a standalone model you can serve like any normal Hugging Face checkpoint with no PEFT dependency at inference.

Whether to merge is a deployment choice. Keeping adapters separate lets you hot-swap tasks on a shared base; merging simplifies the serving stack and can slightly reduce inference overhead. Showing both leaves the reader equipped for either path.

The closing card hands off to post 5, the common mistakes, which catalogs the quiet failure modes — bad data, overfitting, format drift, no eval. You now have a working script; the final post is about not shipping a broken model from it.

The pairing is intentional: code that runs in this post, and the discipline to know whether what it produced is actually good in the next.