LoRA & QLoRA

This post opens the day by reframing fine-tuning around a single surprising fact: you don't have to touch most of a model's weights to change its behavior. The cover sets up LoRA as the technique that made that practical and QLoRA as the variant that made it run on hardware you already own.

The rest of the post installs the core mental model — frozen base, tiny trainable side path, optional 4-bit storage — so that everything in the later mechanics and code posts feels like a consequence rather than a list of magic settings.

The definition deliberately separates the two ideas people tend to blur. LoRA is about *what* you train: instead of all the weights, you train two small matrices that represent the change to a weight, leaving the original frozen. QLoRA is an additive trick about *how you store the frozen part*: in 4-bit instead of 16-bit, so a large model fits in far less memory.

Keeping these distinct matters because you can use LoRA without quantization, and the quantization is what unlocks single-GPU training of the biggest models. Confusing them leads people to think QLoRA is a different algorithm; it's really LoRA plus a memory optimization underneath.

This is the conceptual heart of LoRA. Full fine-tuning learns an update matrix ΔW that has the same shape as the original weight W, which is why it's so expensive. The LoRA hypothesis, supported by the original paper, is that this update is intrinsically low-rank — its useful information lives in a small subspace and doesn't need the full dimensionality of W.

If that's true, you can approximate ΔW as the product of two skinny matrices B and A, where the shared inner dimension r is tiny. You then train only A and B. The frozen W carries all the pretrained knowledge; the small A·B captures just the task-specific adjustment. That single bet is what makes everything else possible.

The comparison makes the practical consequences concrete. Full fine-tuning updates every weight, which means you must store a complete model copy per task, hold optimizer state for all those parameters, and provision large GPU memory. LoRA trains roughly a tenth of a percent to one percent of the parameters, stores adapters measured in megabytes, fits on modest GPUs, and lets you keep many task-specific adapters around the same base.

The swappability point on the right is easy to underrate. Because the base never changes, one frozen model can host many different adapters, which is the seed of the multi-tenant serving story developed in post 2.

This flow diagram shows the one structural change LoRA makes to a layer. The input x still passes through the frozen base weight W as before. In parallel, x also goes through the trainable low-rank path B·A, and the two outputs are added. The layer's result is W·x plus B·A·x.

Visualizing it as a side path clarifies two things: the base computation is untouched, so the model can't catastrophically forget through W, and the adapter is purely additive, which is exactly why it can later be merged back into W or removed without a trace.

Naming A and B explicitly removes the mystery from the rest of the day. A is the down-projection: it takes the full-dimension input and squeezes it to the tiny rank r. B is the up-projection: it expands that r-dimensional signal back to full size so it can be added to W·x. Their product has the shape of W but is built from only 2·d·r numbers instead of d·d.

The initialization detail is worth internalizing now because it explains a slide in post 3: A starts random and B starts at zero. That means at the very first step the adapter contributes nothing and the model behaves exactly like the pretrained base, so training begins from a known-good point rather than from noise.

This snippet compresses the whole idea into the math that matters. W is frozen and square at d-by-d. The adapter is the pair A (r-by-d) and B (d-by-r), and the forward pass adds B·(A·x), scaled by alpha over r. The parameter count line is the punchline: 2·d·r trainable numbers versus d·d for a full fine-tune.

For a concrete sense of scale, if d is 4096 and r is 16, full FT of that one matrix is about 16.8 million parameters while LoRA is about 131 thousand — well under one percent. Multiply across all adapted layers and you see why LoRA's memory and storage footprint is so small.

Here the Q in QLoRA gets its own explanation so it's not mistaken for a separate method. QLoRA keeps the frozen base model in 4-bit NF4 rather than 16-bit, cutting the base's memory by roughly four times. The LoRA adapters you actually train stay in higher precision.

The subtle part is that gradients still need to flow. During the forward and backward pass the 4-bit weights are de-quantized to bfloat16 just in time for each matmul, so the math is done in higher precision even though storage is in 4-bit. The original QLoRA work showed this preserves quality close to full 16-bit fine-tuning, which is why the memory savings come without a large accuracy penalty.

This comparison pins down when to use which. Plain LoRA keeps the base in 16-bit: it's faster per step and simpler to merge, but it uses more VRAM, so the model size you can train is limited by your GPU. QLoRA puts the base in 4-bit NF4: it's somewhat slower per step due to de-quantization but uses dramatically less memory, which is what famously enabled fine-tuning a 65B model on a single 48GB GPU.

The practical rule: reach for plain LoRA when the model already fits comfortably in 16-bit and you want speed; reach for QLoRA when memory is the binding constraint, which is most of the time on consumer hardware.

The plug-in framing turns a technical property into an architectural pattern. Because every adapter is just a few megabytes and the base is frozen and shared, you can treat adapters like configuration: train one per task, version them, and swap them onto the same loaded base without reloading gigabytes of weights.

This is the conceptual seed of multi-tenant serving and of treating each fine-tune as a cheap, ownable asset. It also changes how teams experiment — trying ten variants means storing ten small adapters, not ten full models, which makes iteration nearly free in storage terms.

Closing the concept post with explicit limits prevents the most common overclaims. LoRA, like full fine-tuning, changes behavior and style, not factual knowledge — for fresh or proprietary facts you still want retrieval. It's also not perfectly lossless: a very large behavior change can exceed what a low rank can express, in which case you raise the rank or fall back to full fine-tuning.

And QLoRA's 4-bit base is a memory optimization for training, not a quality upgrade — it trades a little speed and, on the hardest tasks, a sliver of accuracy for the ability to train at all on small hardware. Knowing these boundaries is what keeps you from reaching for LoRA on problems it can't solve.

The closing card points to post 2, which makes the economic case — why these techniques didn't just optimize fine-tuning but changed who can do it at all. This post established the mechanism in concept; the next translates that mechanism into memory, storage, and serving costs.

The day's arc is the familiar one: concept, then why it matters, then mechanics, then code, then pitfalls.