Quantization (4-bit, 8-bit)

Having defined quantization, this post argues why it's one of the most consequential inference techniques in practice. The short version: memory is the budget for running large models, and quantization is how you spend far less of it. That single fact reshapes who can run what, and at what cost.

The emotional hook is real — quantization is the reason an individual can run a 70B model at home at all. That's not a minor optimization; it's a shift in access.

The most important and least intuitive point is that LLM inference is memory-bandwidth-bound, not compute-bound. Generating each token requires reading the full weight set from memory. The GPU's math units often sit idle waiting for data to arrive.

Because of this, cutting the bits per weight cuts the bytes that must be streamed per token, and latency drops close to proportionally. People assume quantization is purely about fitting in memory, but the speedup is just as important — and it comes from the same change.

The bar chart drives home the headline numbers for a 7B model. FP32 needs about 28GB just for weights, FP16 about 14GB, INT8 about 7GB, and INT4 about 3.5GB. Each step down the precision ladder roughly halves the footprint.

Those thresholds map directly onto real hardware. 28GB rules out most consumer cards; 3.5GB fits comfortably on a modest laptop GPU. The chart is really a map of which hardware each precision unlocks.

This slide gives you the formula to compute any of those numbers yourself: weight memory is approximately parameters times bytes-per-parameter. FP32 is 4 bytes, FP16 is 2, INT8 is 1, and INT4 is half a byte.

The reason to internalize this is planning. Before you download a model or rent a GPU, you can estimate in seconds whether it will fit. Remember to add headroom — activations and especially the KV cache for long contexts consume additional memory beyond the weights.

The code turns the formula into a tiny loop that prints the memory for each format. There's nothing fancy here, and that's the point: capacity planning for model serving is genuinely this simple at the first-order level.

Keep this snippet handy. When someone asks 'will this 13B model fit on my 8GB card in 4-bit?' you can answer with arithmetic rather than guessing. The answer, for the record, is yes for the weights — roughly 6.5GB — though you'll want to watch the KV cache.

A subtle but powerful framing: quantization isn't only about shrinking a fixed model, it's about choosing a better model for a fixed memory budget. A 4-bit 13B model occupies similar VRAM to an FP16 7B but is usually noticeably more capable.

This reframes the decision. Instead of 'how small can I make my model,' the question becomes 'given my VRAM, what's the most capable model I can run?' Often the answer is a larger model at lower precision, because you're spending your bits on capability rather than on decimal places you don't need.

The comparison spells out the access story that makes quantization matter beyond raw performance. Without it, large models live on multi-GPU servers, which means cloud rentals, per-hour costs, and your data leaving your machine. With 4-bit, the same models run on one or two local cards.

That changes who gets to participate. Privacy-sensitive use cases become viable because data never leaves the device, costs drop to electricity, and the barrier to experimentation collapses — anyone curious can just try things.

These bullets summarize where the savings concretely land: cheaper hardware because you need less VRAM, faster tokens because less data moves, a small enough footprint for edge and mobile deployment, and local inference that sidesteps both privacy concerns and recurring API bills.

The value is multi-dimensional. Even teams that could afford big GPUs often quantize, because the latency and cost-per-token improvements compound across millions of requests.

The cycle diagram captures why quantization tends to accelerate adoption. Quantizing lets a model fit cheaper hardware, which means you can run it locally without per-call costs, which lowers the friction to iterate, which leads to more usage and experimentation — and the loop reinforces itself.

This flywheel is a big part of why the local-LLM ecosystem exploded: once a capable model fits on hardware people already own, the rate of community tinkering jumps dramatically.

Honesty matters, so this slide names the cost. Quantization trades accuracy for efficiency, and the loss is not uniform. INT8 is usually nearly invisible; aggressive INT4 can measurably degrade reasoning, math, and rare-token handling. Smaller models feel the hit more because they have less redundancy to spare.

The operational rule is simple and non-negotiable: measure quality on your own task, not just the size on disk. A model that loads is not the same as a model that still works, and only an eval tells you which you have.

Closing with a checklist of when quantization is clearly the right call: you're VRAM-constrained, you serve high request volume where per-call savings compound, you need on-device or offline operation, or latency and hosting cost are your actual pain points.

If none of those apply — you have ample memory and low volume — the accuracy trade may not be worth it. But for the vast majority of real deployments, at least one of these is true, which is why quantization is now a default rather than an exotic optimization.

That's the why. Quantization matters because memory is the binding constraint on running large models, and trading a little precision buys enormous gains in cost, speed, and access.

Next we open the hood: the actual equations, the scale-and-zero-point machinery, granularity choices, the outlier problem, and the difference between quantizing after training versus during it.