Quantization (4-bit, 8-bit)

This post is the mechanics — the part that demystifies the scary-sounding '4-bit.' The reassuring thesis is that quantization is mostly rounding with a well-chosen ruler. Once you see the equations and the handful of design choices, the black box opens up.

We'll cover the quantize/dequantize math, symmetric versus affine mapping, how finely you group the scales, the outlier problem that dominates LLM quantization in practice, and the choice between post-training quantization and quantization-aware training.

The two core equations are worth memorizing. Quantize: q = round(x / scale) + zero_point, then clamp to the integer range. Dequantize: x̂ = scale × (q − zero_point). The integer q is what gets stored and computed on; the scale and zero-point are the small metadata that translate to and from real values.

The difference x − x̂ is the quantization error, and every technique in this space is ultimately about minimizing that error for a given bit budget. Notice the error comes from two places: rounding to the nearest grid point, and clamping anything outside the chosen range.

The flow diagram traces the round trip for a single weight: start with the float x, divide by the scale and round to get an integer q, store q in 4 or 8 bits, and later multiply by the scale to recover x̂, an approximation of the original.

Seeing it as a loop makes the lossiness obvious and intuitive. You're snapping continuous values onto a discrete grid; the finer the grid (more bits) or the tighter the range (better scale), the closer x̂ lands to x.

Symmetric versus affine is the first real design choice. Symmetric quantization fixes the zero-point at 0 and centers the range on zero, with scale = max|x| / qmax. It's simple and ideal for weights, which are roughly symmetric around zero.

Affine (asymmetric) quantization adds a nonzero zero-point so the integer range can map onto a shifted or skewed distribution, using the full range of codes. This suits activations like post-ReLU values that are all positive. Choosing the right one per tensor type squeezes out avoidable error for free.

Granularity — how many scales you use — is arguably the most impactful knob. One scale for an entire tensor is cheap to store but dangerously fragile: a single outlier weight stretches the range, forcing a large scale that quantizes all the ordinary weights coarsely.

The fix is to use more scales: per-channel (one per row/column) or per-group (one per block of, say, 64 or 128 weights). Each block gets its own ruler, so an outlier only degrades its own small neighborhood. The extra storage is a tiny fraction of the savings, and the error reduction is large — which is why group-wise 4-bit is the modern default.

This comparison makes the granularity trade-off concrete. Per-tensor uses one scale for everything: smallest metadata, but a single outlier hurts every weight and error is highest. Per-group with a group size of 128 uses one scale per 128 weights: outliers stay local, the extra storage is negligible, and error drops substantially.

The practical lesson is that at 4-bit you almost always want per-group. Per-tensor might be acceptable at 8-bit where you have more headroom, but for aggressive quantization, fine granularity is what keeps the model coherent.

Outliers are the single biggest practical obstacle in LLM quantization, which is why they get their own slide. In large transformers, a small number of activation values can be 10 to 100 times larger than the rest. A naive scheme lets these dominate the range, crushing precision for the overwhelming majority of normal values.

This is the entire motivation behind the famous methods: LLM.int8() detects outlier dimensions and keeps them in FP16; GPTQ uses second-order information to compensate as it quantizes; AWQ identifies and protects the most salient weight channels. They are all, at heart, outlier-management strategies.

The code generalizes the earlier hand-quantization into a per-group symmetric INT4 quantizer. We reshape the weights into rows of 128, compute one scale per row as the row's max magnitude divided by qmax (7 for signed INT4), then round and clamp within each group independently.

Returning both the integers and the per-group scales is the key detail — you need both to dequantize. This twelve-line function is essentially a stripped-down version of what production quantizers do; the real ones add smarter range selection and error compensation, but the grouping skeleton is exactly this.

These steps describe how a real quantizer runs end to end. First you choose the configuration: bit width, granularity, and symmetric versus affine. Then you push a small calibration dataset through the model to observe the actual ranges of weights and activations. From those you compute scales and any zero-points, round the weights onto the integer grid, and pack everything into a compact file with its metadata.

Calibration is the step beginners overlook. The scales are only as good as the data used to estimate the ranges, which is why a later post devotes a mistake to bad calibration data.

PTQ versus QAT is the last major fork. Post-training quantization takes an already-trained model and quantizes it using only a little calibration data; it runs in minutes and includes GPTQ, AWQ, and LLM.int8(). It's the default for most people because it's cheap and usually good enough.

Quantization-aware training simulates the quantization error during training (or fine-tuning) so the model learns weights that are robust to it. It costs a training run but recovers the most accuracy, especially at very low bit widths. The rule of thumb: reach for QAT only when PTQ's accuracy at your target bit width isn't acceptable.

The decision tree turns that fork into an actionable choice. If you have a training pipeline and budget and you're pushing to 4 bits or below, QAT buys you the best accuracy. If you have the budget but only need 8-bit, PTQ is plenty. And if you have no training pipeline, PTQ methods like GPTQ or AWQ are the pragmatic answer.

Most practitioners live in the PTQ branches. QAT is powerful but the cost is real, so it's reserved for cases where every point of accuracy at extreme compression matters.

That's the machinery: equations, mapping types, granularity, outliers, and the PTQ/QAT choice. With these knobs in view, quantization stops being magic and becomes a set of deliberate trade-offs you can reason about.

Next we get hands-on — loading a real model in 8-bit and 4-bit, configuring NF4, running a pre-quantized GPTQ checkpoint, and measuring the VRAM you actually save.