Quantization (4-bit, 8-bit)

Quantization is one of those techniques that sounds intimidating until you see what it actually is: storing the model's numbers using fewer bits. This post is the conceptual foundation for the rest of the day, so the goal is to make the idea click before we touch math or code.

The one-sentence version is that you swap high-precision floats for low-precision integers plus a tiny amount of bookkeeping. Everything else — formats, schemes, tools — is detail on top of that single trade.

The definition deliberately separates two things that beginners conflate: the size of the model (number of parameters) and the precision of each parameter (bits per weight). Quantization changes only the second. A 7B model is still 7 billion weights after quantization; each weight is just described with fewer bits.

That distinction matters because it tells you what quantization can and can't do. It won't make a model smarter or teach it new facts. It makes the same model cheaper to store and faster to run, with a small, usually acceptable cost in numerical fidelity.

The reason to care comes down to a fact about LLM inference: it is memory-bound, not compute-bound. To generate each token, the hardware must stream the entire set of weights out of memory. The arithmetic is cheap; moving the bytes is the bottleneck.

So halving the bits per weight roughly halves the bytes moved per token, which on typical GPUs translates almost directly into more speed and the ability to fit the model in less VRAM. This is why a technique that 'only' changes storage has such an outsized effect on real-world performance.

This slide lays out the number formats you'll meet constantly. FP32 is the classic four-byte float; FP16 and BF16 are the two-byte formats most models train and ship in. INT8 packs each weight into a single byte, and INT4 into half a byte — eight times smaller than FP32.

The left/right split also signals intent: the high-precision formats are about training and exactness, while the integer formats are primarily an inference optimization. Knowing which column you're in tells you what trade-offs are in play.

Here is the trick that makes the whole thing work, and the single most useful mental model in the post. A quantized weight is not the real number — it's an index into a small set of possible values. To recover the real value you multiply by a stored scale and, in the affine case, subtract a zero-point.

Because you only need a handful of these floats per group of weights, the metadata is negligible while the bulk of the data shrinks dramatically. If you remember one thing from this post, make it 'real ≈ scale × (int − zero_point)'.

The flow diagram shows the mechanical path from float weights to stored integers: look at the values in a group, find their range, scale and round them into the integer grid, then store the integers alongside the scale. At inference you reverse the last step to get an approximation of the original.

The key idea to absorb is that quantization is fundamentally a rounding operation with a well-chosen ruler. The cleverness in real methods is almost entirely about how you choose that ruler and how finely you group the weights.

It's reasonable to be suspicious that rounding billions of weights could possibly be harmless. The answer is that neural networks are massively over-parameterized and trained with noisy gradients, dropout, and other perturbations, so they are inherently robust to small disturbances.

The information a network encodes lives in the collective pattern of its weights, not in the precise low-order bits of any individual one. Rounding each weight slightly is just one more small perturbation, and a well-trained network absorbs it with barely a change in behavior — which is exactly why 8-bit is often indistinguishable from full precision.

This code makes the abstract concrete: it quantizes a tensor to INT8 by hand. We pick the signed INT8 range (±127), compute a single scale as the largest magnitude divided by 127, then divide, round, and clamp to get integers. Dequantizing is just multiplying back by the scale.

The printed mean error will be tiny relative to the weight magnitudes, which is the whole point. This is per-tensor symmetric quantization — the simplest possible scheme — and the later posts build on exactly this skeleton with grouping and smarter range selection.

Not everything in a model is quantized equally. Weights are the easy, high-value target and are almost always quantized first. Activations are harder because they vary per input and contain extreme outliers. The KV cache becomes a major memory consumer at long context lengths and is increasingly quantized too.

The practical wisdom is to keep sensitive components — often embeddings and the final output head — in higher precision while aggressively quantizing the bulk of the transformer layers. This selective approach captures most of the savings with the least quality risk.

It's worth situating quantization against the other model-compression techniques so you don't confuse them. Quantization keeps every weight but describes each with fewer bits, and done well it's close to lossless. Pruning removes weights or whole neurons, and distillation trains a smaller 'student' model to imitate a larger one.

The difference that matters in practice: quantization is mostly a post-hoc inference trick you can apply to an existing checkpoint in minutes, while pruning and distillation change the architecture and usually require retraining. They're complementary — you can quantize a distilled model — but they solve the problem in different ways.

Finally, the pipeline diagram places quantization in the model lifecycle. Training happens in FP16 or BF16 because gradients need the precision. Once training is done, you quantize the finished weights to INT8 or INT4, pack the integers and their scales into a compact file, and serve that.

The takeaway is that quantization sits cleanly after training and before deployment, which is why it's such a popular optimization: you don't have to touch your training setup to benefit from it.

That wraps the concept. You now have the core mental model — fewer bits per weight, recovered via a stored scale and zero-point, tolerated because networks are robust.

Next we make the case for why this matters so much in practice, with the actual memory and cost numbers that turn quantization from a neat trick into the thing that put capable LLMs on consumer hardware.