LoRA & QLoRA

This is the hands-on post, so the cover makes a concrete promise: a complete QLoRA fine-tune that runs on a single 16GB GPU. The aim is that a reader can copy these snippets in order, point them at their own JSONL data, and finish with a working tuned adapter the same evening.

Everything uses the standard QLoRA recipe established in the prior posts — a 4-bit NF4 base with LoRA adapters on the attention layers — because that's the workflow that actually fits accessible hardware and is what most teams ship.

The overview slide gives the four stages so each code slide has a clear home: load a 7B model in 4-bit, prepare it for k-bit training, attach LoRA adapters, then train, save the adapter, and run inference. Naming the sequence up front turns the code that follows into a map a reader can navigate rather than a wall of snippets.

This ordering deliberately mirrors the pipeline diagram later in the post, so the same workflow is reinforced once as a checklist and once visually.

Step one loads the base model in 4-bit, which is what makes a 7B model fit in roughly 5GB. The BitsAndBytesConfig requests nf4 quantization, enables double quantization to squeeze the constants, and sets bfloat16 as the compute dtype used when de-quantizing for matmuls — exactly the data-type choices explained in post 3.

The final line sets the pad token to the eos token, which matters for causal models that lack a dedicated pad token; without it, batching can throw confusing errors. Using Mistral-7B here is incidental — swap in any compatible base and the rest of the script is unchanged.

Step two contains the line most beginners omit: prepare_model_for_kbit_training. It does quiet but important work — enabling gradient checkpointing to save activation memory and casting layer norms and embeddings to a stable dtype so 4-bit training doesn't diverge. Skipping it is a leading cause of NaNs and silent quality loss, which is why the mistakes post calls it out again.

The LoraConfig then attaches adapters to all four attention projections — query, key, value, and output — with rank 16 and alpha 32. The print_trainable_parameters call confirms you're training only a fraction of a percent of the model, which is the reassurance that the setup is working as intended.

Step three formats the dataset, and this is where most of the quality is won or lost. The to_text function wraps each record in a fixed instruction/response template and appends the eos token after the response so the model learns where to stop generating — an omission that produces models that ramble forever.

The template itself is arbitrary, but consistency is not negotiable: every row must use the identical structure, and the same template must be reused at inference time. Loading from JSONL is the convention for instruction data because each line is one independent example, which keeps the dataset easy to inspect and append to.

Step four runs the actual training and is where the QLoRA-specific optimizer choice appears. The SFTConfig sets two epochs, a small per-device batch of two with gradient accumulation of eight (an effective batch of sixteen), a 2e-4 learning rate, and bf16. The optim argument selects paged_adamw_8bit — the paged optimizer from post 3 that survives transient memory spikes.

The SFTTrainer from TRL wires the model, dataset, and arguments together and handles tokenization and the loop. The closing save_model writes only the LoRA adapter, typically tens of megabytes, which is why fine-tunes here are cheap to store and trivial to version compared to full model checkpoints.

The pipeline diagram restates the entire workflow visually so the code slides cohere into one picture: load the 4-bit NF4 base, prepare and attach LoRA, format the instruction pairs, train with the paged AdamW optimizer, then load and infer. Placing it right after the training code helps a reader zoom back out from line-level detail to the overall shape before moving to inference.

This is the same five-stage flow named in the overview slide, now drawn — the repetition is intentional reinforcement, not filler.

Step five shows inference done the way you'd actually serve it: reload the base model — here again in 4-bit — and attach the saved adapter with PeftModel, rather than reusing the in-memory training object. The prompt uses the exact same template as training, with the same headers and spacing, which is essential because a format mismatch silently tanks quality.

Decoding with skip_special_tokens cleans up the output. This step closes the loop the whole post built toward: the adapter you trained is now answering a real prompt, and the formatting discipline from step three pays off directly here.

The 'lines that matter' slide extracts the four details that most affect results so they don't get buried in the code. nf4 plus double quantization is what fits a 7B model in roughly 5GB. prepare_model_for_kbit_training is what keeps 4-bit training stable. The paged AdamW optimizer is what lets a long run survive memory spikes. And save_model storing only the adapter is what keeps each fine-tune lightweight.

These four are precisely the points that separate a script that merely runs from one that produces a good, deployable model — they're the difference-makers worth memorizing.

The optional merge slide shows how to produce a standalone model for deployment, and it deliberately reloads the base in fp16 rather than 4-bit before folding in the adapter. That ordering is not a stylistic choice — it's the correct path, because merging through a 4-bit base is lossy. After merge_and_unload, you have a normal Hugging Face checkpoint you can serve without any PEFT dependency.

Whether to merge is a deployment decision: keeping the adapter separate lets you hot-swap tasks on a shared base, while merging simplifies the serving stack. Showing the fp16 reload here sets up the dedicated mistake slide that follows.

This mistake slide isolates the single most common QLoRA deployment bug: trying to merge an adapter into a base that's still loaded in 4-bit. The quantization makes the fold lossy, so the merged weights come out subtly corrupted and the model is quietly worse — with no error to warn you.

The fix is exactly what the previous slide demonstrated: reload the base in fp16 or bf16 first, attach the adapter, then merge_and_unload. Calling this out as its own slide, right after showing the correct code, is meant to burn the rule in before a reader hits it in production.

The closing card hands off to post 5, the common mistakes, which catalogs the quiet failure modes specific to LoRA and QLoRA — wrong rank, alpha-rank mismatch, skipping k-bit prep, the lossy merge, and 4-bit inference when you needed speed. You now have a working script; the final post is about ensuring what it produced is actually good.

The pairing is intentional: runnable code here, and the judgment to validate its output next.