Quantization (4-bit, 8-bit)

This is the hands-on post, and it's deliberately code-heavy. The aim is that you can copy these snippets, run them on a GPU, and watch a 7B model that needed 14GB drop to under 4GB. Seeing the gigabytes disappear in your own terminal does more to build intuition than any diagram.

We'll use the Hugging Face Transformers + bitsandbytes stack for on-the-fly quantization, then show the pre-quantized GPTQ path that skips the quantization step entirely.

First, the setup. The stack is Transformers for the model API, Accelerate for device placement, and bitsandbytes for the actual INT8 and 4-bit kernels. The important caveat lives in the comment: bitsandbytes 4-bit support targets Linux (or WSL) with an NVIDIA GPU.

If you're on macOS or CPU-only, this exact path won't work and you should use llama.cpp with GGUF files instead — a note we repeat later because it trips people up constantly. The imports themselves are the standard trio plus BitsAndBytesConfig, which carries the 4-bit settings.

The 8-bit load is almost anticlimactic: pass load_in_8bit=True and device_map="auto", and Transformers quantizes the weights to INT8 as they load. A 7B model that's ~14GB in FP16 lands around 7GB on the GPU.

This is the gentlest form of quantization and a great default first step. INT8 is usually nearly indistinguishable from FP16 in quality, so if 8-bit fits your memory budget and you just want easy savings with minimal risk, you can often stop right here.

The 4-bit load needs a small config object but unlocks the big savings. The four settings matter: load_in_4bit turns it on; nf4 selects the NormalFloat-4 datatype; the bfloat16 compute dtype means matmuls happen in bf16 after dequantizing on the fly; and double quantization compresses the scale factors too. This is precisely the QLoRA loading recipe.

After this, the same 7B model sits around 3.5-4GB on the GPU. The compute dtype detail is worth understanding: the weights are stored in 4-bit but are dequantized to bf16 just-in-time for each matrix multiply, so you keep low storage without doing arithmetic in 4-bit.

This slide explains the two ingredients that make 4-bit work as well as it does. NF4 is a datatype whose 16 representable levels are spaced to match the roughly normal distribution of neural-net weights, so it allocates precision where the weights actually cluster instead of wasting codes on empty tails like plain INT4 would.

Double quantization then quantizes the scale factors themselves — there's one scale per group, and those scales are floats, so compressing them saves roughly another 0.4 bits per weight on average. Neither is exotic; together they're the reason QLoRA's 4-bit models stay so close to full precision.

Generation from the 4-bit model is identical to a normal model — the quantization is invisible at the API level. You tokenize the prompt, call generate, and decode. The dequantization happens inside the forward pass automatically.

The comment makes the key empirical point: quality stays close to FP16 for an NF4-loaded 7B. You'd need careful evals to see the difference on most tasks, which is exactly why 4-bit became the practical default for running models locally. Greedy decoding (do_sample=False) is used here for reproducible output.

Measuring is the whole point of the post, so this snippet reads torch.cuda.max_memory_allocated to report peak GPU memory. The commented numbers — roughly 14GB FP16, 7.2GB at 8-bit, 4.1GB at 4-bit — are representative for a 7B model and turn the savings from a claim into a measurement.

Get in the habit of printing this whenever you load a quantized model. It catches surprises (an unquantized layer, an oversized KV cache) immediately, and it's the honest way to verify that your config did what you intended rather than trusting the label.

Not all quantization happens at load time. This snippet loads a pre-quantized GPTQ checkpoint — the weights are already 4-bit on disk, so there's no bitsandbytes step and the download itself is roughly 4GB instead of 14GB. Transformers detects the GPTQ config in the checkpoint and wires up the right kernels automatically.

This is often the easiest path in practice: someone has already done the quantization and published the result, so you just download and run. GPTQ and AWQ checkpoints are abundant on the Hub, and because the quantization is baked in, loading is fast and deterministic.

The pipeline diagram summarizes what the loader is doing on your behalf regardless of path: it reads the weights (FP16 or a pre-quantized file), quantizes if needed, places shards on the GPU according to device_map, and runs with on-the-fly dequantization during the forward pass.

The value of seeing this is that the one-line convenience hides several real steps. When something goes wrong — out of memory, a placement error, a missing kernel — knowing these stages tells you where to look.

These practical notes collect the gotchas that waste the most time. bitsandbytes 4-bit really does want Linux and NVIDIA. Pre-quantized GPTQ/AWQ files skip the load-time quantize step. On CPU or Mac, llama.cpp with GGUF is the right tool. And because this ecosystem moves fast, pinning versions of transformers, accelerate, bitsandbytes, and auto-gptq saves you from confusing breakage.

None of these are deep, but each one is a half-day of frustration if you learn it the hard way instead of from a list.

The final content slide flags the most common conceptual gotcha: you cannot directly full-fine-tune a bitsandbytes 4-bit model, because the base weights are frozen integers, not trainable floats. Calling .train() and expecting the quantized weights to update will fail or do nothing useful.

The correct pattern is to attach small LoRA adapters on top of the frozen 4-bit base and train only those — which is exactly what QLoRA does, and why it pairs naturally with the NF4 config shown earlier. This connects the day's quantization material back to the fine-tuning material from Day 61-62.

That's quantization in practice: a few config lines to load in 8-bit or 4-bit, NF4 and double-quant for the aggressive case, pre-quantized GPTQ for zero-effort loading, and always measuring the VRAM you saved.

Next we close the day with the failure modes — the subtle mistakes that turn a near-lossless quantized model into a quietly broken one, and exactly how to avoid each.