LoRA & QLoRA

This post turns LoRA and QLoRA from concepts into concrete mechanics, so the hyperparameters in the code post stop feeling like incantations. The cover frames it as clever bookkeeping about where the trainable parameters live, plus a quantization layer underneath.

If you finish this post knowing what rank and alpha control, why B starts at zero, which modules to target, and what NF4 and paged optimizers do, then the code in post 4 is wiring you understand rather than settings you copy.

This slide states the forward pass with full precision so there's no ambiguity. For a frozen weight W and input x, LoRA computes h equals W·x plus the scaled adapter term, (alpha over r) times B times (A times x). A has shape r-by-d, B has shape d-by-r, and r is much smaller than d. Only A and B receive gradients.

The scaling factor alpha over r is easy to gloss over but matters: it normalizes the adapter's contribution relative to its rank, so that increasing capacity by raising r doesn't simultaneously blow up the magnitude of the update. Understanding this exact expression is what makes the alpha and rank discussions that follow precise rather than hand-wavy.

The network diagram visualizes the low-rank bottleneck that defines LoRA. The input dimension d feeds into a narrow middle layer of width r, which then expands back out to d. That pinch in the middle is the rank, and it's the entire reason the adapter has so few parameters.

Seeing it drawn this way makes the trade-off intuitive: a wider middle (larger r) can carry more information about the update but costs more parameters, while a narrower middle is cheaper but can express less. The whole tuning game for capacity happens at that one bottleneck.

This slide gives rank its proper role as the main capacity dial. r sets how much information the update can carry. A small rank of 4 to 8 is cheap and is usually plenty for tasks like adjusting tone, enforcing a format, or a narrow skill. Larger ranks of 16 to 64 give the adapter room to express bigger behavior shifts.

The cost of larger r is more memory and a greater tendency to overfit, especially on small datasets. The practical mindset is to treat r as a hyperparameter to sweep against a validation set rather than a value to fix once — the right rank is the smallest one that captures the behavior you need.

Alpha is the dial people most often misunderstand, so it gets dedicated treatment. The adapter's output is multiplied by alpha over r, which means alpha controls the effective strength of the update independently of how many parameters it has. This decoupling is deliberate: it lets you change the rank for capacity reasons without having to re-tune the update's magnitude from scratch.

The common heuristic of setting alpha to twice the rank is a reasonable starting point, but it's just a heuristic. The deeper takeaway, reinforced in the mistakes post, is that what actually matters is the ratio alpha over r, and it's worth treating alpha as a genuine hyperparameter rather than a fixed convention.

This snippet explains the initialization choice that makes LoRA train smoothly. A is initialized with small random values and B is initialized to all zeros. Because the adapter output is B times (A times x), and B is zero, the entire adapter contributes nothing on the first step.

That means training starts from h equals W·x exactly — the unmodified pretrained model — rather than from a randomly perturbed version of it. As gradients flow, B moves away from zero and the adapter begins to take effect. Starting from the known-good base instead of from noise is a large part of why LoRA fine-tuning is stable and rarely needs warmup gymnastics.

Choosing which modules to adapt is a real lever, and this slide lays out the ladder. Targeting the attention query and value projections (q_proj and v_proj) is the classic, well-tested default and is enough for many tasks. Adding the key and output projections (k_proj, o_proj) gives the adapter more places to act.

Going further to the MLP projections — up, down, and gate — enables larger behavior shifts but adds substantially more trainable parameters and memory. The guidance, expanded in the mistakes post, is to broaden the target set before reflexively cranking the rank, since more insertion points can help capacity in a different way than a wider bottleneck does.

The stack diagram makes QLoRA's memory architecture tangible by showing the three layers that coexist during training. At the top sit the LoRA adapters in bf16 — small and the only trainable part. Beneath them is the frozen base in 4-bit NF4, holding all the pretrained knowledge at a quarter of the usual memory. Underneath both sits the paged optimizer, which can spill state between GPU and CPU during memory spikes.

Laying it out as a stack clarifies that these are three distinct optimizations addressing three distinct costs: adapters cut the trainable-parameter count, NF4 cuts the frozen-weight storage, and paging handles transient peaks. QLoRA is the combination, not any single one.

This slide explains the two quantization tricks that make QLoRA's 4-bit base work without wrecking quality. NF4, or 4-bit NormalFloat, is a data type designed for the roughly normal distribution of neural network weights, so it allocates its limited levels where the weights actually concentrate — losing less than naive int4. Double quantization then quantizes the quantization constants themselves, squeezing out a little more memory.

Crucially, the 4-bit weights are de-quantized to bf16 just in time for each matrix multiply, so the actual arithmetic happens in higher precision. This is why QLoRA can store the base so compactly yet still produce gradients accurate enough to match near-full-precision fine-tuning.

This config snippet ties the whole post's theory to the exact knobs you'll set. The BitsAndBytesConfig turns on 4-bit loading, selects the nf4 quant type, enables double quantization of the constants, and sets bfloat16 as the compute dtype used when weights are de-quantized for matmuls. The LoraConfig sets rank 16 with alpha 32 — the alpha-equals-twice-rank heuristic — targets the query and value projections, and applies a small dropout.

Every argument here maps directly to a concept from earlier slides, which is the point: by this stage nothing in the configuration should be mysterious. You can read each line and say what it controls and why that value is reasonable.

The final mechanics slide covers paged optimizers, which solve a problem that only shows up on long runs. GPU memory use during training isn't flat; gradient computation produces transient spikes, and one bad spike can exceed your VRAM and crash a job hours in. Paged optimizers, borrowed conceptually from operating-system memory paging, move optimizer state between GPU and CPU during these peaks.

The effect is robustness: instead of provisioning for the worst-case momentary spike, you let the optimizer page out under pressure and continue. In practice this is what lets a tight QLoRA run on a memory-constrained GPU survive to completion rather than dying at hour three.

The closing card hands off to post 4, which assembles every concept here into one runnable QLoRA script. You've now seen the forward pass, rank and alpha, the zero-init trick, target modules, and the NF4 and paging machinery — the next post is about wiring them together correctly.

With the mechanics understood, the code becomes a matter of arranging known pieces rather than trusting unfamiliar settings.