LoRA & QLoRA

This closing post is about the LoRA and QLoRA failures that don't announce themselves with an error. A run can train cleanly, show a falling loss, finish without a crash — and still produce a mediocre model. The cover names that uncomfortable reality directly.

The goal is to inoculate readers against the specific, common traps so they stop paying for them in wasted GPU hours and shipped regressions. Each mistake here is one a careful practitioner is likely to hit at least once, and each comes with the habit that catches it.

The first trap is setting the rank wrong for the task. If the rank is too low, a large behavior shift simply can't fit — the low-rank bottleneck physically cannot express the change, and the model underfits no matter how long you train. If the rank is pointlessly high, you waste memory and invite overfitting, especially on small datasets where the extra capacity just memorizes examples.

The defense is to match rank to ambition: roughly 4 to 8 for tone and format tasks, 16 to 32 for richer skills, and to sweep the value against a validation set rather than picking one number and hoping. Rank is the main capacity dial, and treating it as a fixed constant is how runs quietly underperform.

The second trap is the most insidious because it hides in the scaling math. The adapter's effect is multiplied by alpha over r, so when you raise the rank to add capacity but leave alpha unchanged, you silently halve the adapter's effective strength. People do exactly this — bump r expecting more, and get a weaker update instead.

The fix is to think in terms of the alpha-over-r ratio, not the raw numbers. If you change rank, adjust alpha to keep the ratio where you want it, or deliberately re-tune both. Tracking the ratio turns a confusing 'why did more capacity make it worse' mystery into a controlled decision.

The bar chart makes the rank trade-off concrete on a mid-size task. A rank of 4 does fine for light tasks but leaves quality on the table for something richer. A rank of 16 is the balanced default that captures the behavior without excess capacity. A rank of 128 actually scores worse here because, on a small dataset, all that extra capacity overfits.

The numbers are illustrative, but the inverted-U shape is the real lesson: more rank is not monotonically better. The best rank is the smallest one that fully captures your target behavior, and going past it costs both memory and generalization.

The third trap is QLoRA-specific: skipping prepare_model_for_kbit_training. Without it, layer norms and embeddings are left in a state that makes 4-bit training unstable — you get loss spikes, NaNs, or a model that trains but is quietly worse. The function also enables gradient checkpointing, which you generally want for memory.

It's a single line, and its absence is one of the most common reasons a QLoRA run misbehaves. The habit is simple: in any k-bit training script, call it immediately after loading the quantized model and before attaching the adapter, every time, without thinking about whether you can skip it.

This snippet operationalizes QLoRA stability into a short checklist. Call prepare_model_for_kbit_training first. Set use_cache to False, which is required when gradient checkpointing is on or you'll get warnings and incorrect behavior. Use a bf16 compute dtype and a paged optimizer. And if you see NaNs, the first two things to check are lowering the learning rate and confirming the prep line actually ran.

The assert on is_gradient_checkpointing is a cheap guard that catches the case where the prep step was silently skipped. Wiring these checks in from the start converts 'my QLoRA run is unstable' from a debugging session into a setup you can trust.

The fourth trap is under-targeting: adapting only the query and value projections and then being surprised when quality plateaus on a harder task. That default is fine for many tasks, but it gives the adapter only two places per attention block to act, which can bottleneck capacity in a way that raising rank doesn't fully fix.

The guidance is ordered deliberately: if quality stalls and you have memory headroom, broaden the target modules — add the key and output projections, then the MLP projections — before reflexively cranking the rank. More insertion points and a wider bottleneck are different levers, and broadening targets is often the more effective one for richer behavior changes.

The fifth trap is the lossy 4-bit merge, and it deserves emphasis because it produces a silently degraded model with no error. Calling merge_and_unload while the base is still loaded in 4-bit tries to fold the adapter through the quantization, which can't be done cleanly — the resulting weights are corrupted relative to what you trained.

The correct path is to reload the base in fp16 or bf16, attach the adapter, then merge. This is the single most common QLoRA deployment bug, and because the training and the merge both 'succeed,' it often ships before anyone notices the quality drop. The rule is unconditional: never merge through a 4-bit base.

The decision diagram turns the merge question into a quick flowchart. If you don't need a standalone model, keep the adapter separate and load it with PEFT — the simplest and safest option. If you do want a merged model, the next question is whether the base is currently loaded in 4-bit: if yes, reload it in fp16 first and then merge; if it's already in fp16, merge_and_unload is safe to call directly.

Framing it as a decision tree makes the one dangerous path — merging through 4-bit — impossible to wander into by accident, because the diagram forces you to check the base's precision before merging.

The sixth trap is a precision-versus-speed confusion at serving time. QLoRA's 4-bit base is a training-memory optimization; it does not make inference faster, and often makes it slower per token because every matmul pays a de-quantization cost. Teams sometimes assume their QLoRA model is also a fast inference model and are surprised by the latency.

The fix depends on the goal: if you need fast serving, merge the adapter into an fp16 model and serve that, or apply an inference-oriented quantization like GPTQ or AWQ chosen specifically for speed. The general principle is that the quantization that's best for training memory and the one that's best for inference latency are different decisions, made for different reasons.

This snippet sketches the sanity gate every fine-tune should pass before shipping. Score the prompted baseline and the tuned model on the same fixed evaluation cases, then assert the tuned model actually beats the baseline — if it doesn't, you haven't gained anything and shouldn't deploy. The comment adds a cheap manual check: eyeball a few outputs to confirm the model learned to stop at the EOS token, since failure to stop is a classic symptom of a formatting or EOS mistake.

The grading function is task-specific — exact match, a rubric, or an LLM judge — but the gate pattern is universal. Making 'beat the baseline on a fixed eval' an enforced assertion rather than a hope is what keeps regressions from reaching users.

The pre-flight checklist consolidates the post into five habits: sweep the rank while tracking the alpha-over-r ratio, always run prepare_model_for_kbit_training in QLoRA, broaden target modules before maxing out rank, reload in fp16 before merging, and beat the prompted baseline on a fixed eval set. Run through it before and after every fine-tune.

Notice that these are mostly about process and measurement, not modeling cleverness. That's the real lesson of the day: getting good results from LoRA and QLoRA is less about exotic settings and more about choosing sane defaults, avoiding a handful of silent traps, and proving the result is actually better.

The closing card wraps the entire day. With the concept, the economics, the mechanics, a full runnable script, and the common pitfalls covered, you can now run LoRA and QLoRA end to end — train an adapter on a single GPU, merge it correctly, and validate that it genuinely helped.

That last part is the point. The most valuable skill here isn't launching the trainer; it's the judgment to pick the right rank and precision for the job and to verify, against a real baseline, that your fine-tune earned its place in production.