Quantization (4-bit, 8-bit)

The final post of the day is about failure modes, because the gap between 'quantization works great' and 'quantization ruined my model' is almost always a process mistake, not a flaw in the technique. The five traps here are the ones that bite real teams.

The unifying theme is humility about measurement: quantization damage is subtle and easy to miss, so the antidote to every mistake on this list is to measure rather than assume.

Mistake one is the most seductive: judging success by file size and a single happy-path prompt. The model is 4GB now, it loads, it answered your test question — so it must be fine. But quantization degradation rarely shows up there.

It shows up in the tails: slightly worse arithmetic, weaker recall over long contexts, a few more hallucinations, subtle reasoning slips. None of these are visible from disk size or one cherry-picked prompt, which is exactly why people ship quietly degraded models and only notice when users complain.

The fix is disciplined evaluation. Run the same eval set on the FP16 baseline and the quantized version and compare. Track task accuracy rather than only perplexity, because perplexity can look fine while downstream task performance slips. Deliberately include the hard cases — math, long context, code — where quantization damage concentrates.

Then set a threshold in advance: the maximum accuracy drop you'll accept. Deciding the bar before you see the numbers keeps you honest and turns a vibes-based call into an engineering one.

Mistake two is bad calibration data, which is specific to PTQ methods like GPTQ and AWQ. These methods estimate activation ranges from a small calibration set, and the resulting scales are only as representative as that data. Calibrate on generic English web text and then deploy on medical, legal, or code traffic, and the scales are tuned for the wrong distribution.

The symptom is frustrating: the model looks fine on generic prompts but underperforms on your actual workload. Because the cause is buried in the quantization step, it's easy to misdiagnose as a model problem rather than a calibration problem.

The fix is to calibrate on text that resembles your real inputs. The snippet pulls a couple hundred samples from a domain-appropriate corpus and passes them as calibration data to the quantizer. A few hundred representative samples is usually enough; thousands of the wrong samples are worse than hundreds of the right ones.

The comment states the principle bluntly: generic web text is not your specialized traffic. If you serve a narrow domain, your calibration set should look like that domain, full stop.

Mistake three is over-quantizing — forcing aggressive low-bit quantization onto models or layers that can't take it. Small models in the 1-3B range have far less redundancy than a 70B, so they degrade noticeably faster at 4-bit. And within any model, certain components are sensitive: the embedding table, the final LM head, and normalization layers.

Uniformly slamming everything to INT4 is how you turn a coherent model into a confused one. The damage is uneven, so a blunt one-size-fits-all setting is almost always the wrong choice for aggressive quantization.

The fix is selectivity. For small models under about 3B, prefer INT8, where the quality hit is minimal. Keep the output head and embeddings in higher precision since they're disproportionately sensitive. Use per-group rather than per-tensor scales when you do go to 4-bit. In general, a mixed-precision scheme — most layers at 4-bit, the sensitive few higher — beats uniform INT4 at almost no extra cost.

The meta-point is that quantization is not all-or-nothing. The best results come from spending your precision budget where it matters and economizing where it doesn't.

Mistake four is treating activations like weights. Weights are relatively well-behaved and quantize cleanly. Activations contain extreme, input-dependent outliers that naive uniform INT8 simply cannot represent without destroying precision for the normal values. Applying the same scheme to both is a classic and damaging error.

This is the exact problem LLM.int8() solves by keeping outlier dimensions in FP16, and it's a major reason weight-only quantization (leaving activations in higher precision) is so popular for LLMs. If you must quantize activations, you need an outlier-aware method, not the vanilla approach you used for weights.

The comparison makes the weights-versus-activations distinction concrete. Weights are fairly well-behaved, quantize cleanly to 4-bit, are static after training, and are the easy win. Activations have large rare outliers, are harder to quantize, change with every input, and must be handled with care.

The practical implication is the popularity of weight-only schemes: quantizing the weights captures most of the memory savings while sidestepping the activation-outlier minefield entirely. When you see 'W4A16' (4-bit weights, 16-bit activations), this is exactly the trade-off being made.

Mistake five is comparing apples to oranges across formats. '4-bit' is not a single, well-defined thing. GPTQ, AWQ, bitsandbytes NF4, and GGUF's Q4_K_M differ in group size, datatype, which components stay in high precision, and the error-compensation algorithm. Two models both labeled '4-bit' can behave quite differently.

The second half of this trap is portability: a format that loads on your NVIDIA GPU (like bitsandbytes 4-bit) may not run elsewhere, while GGUF targets CPU and Apple Silicon. Assuming '4-bit is 4-bit' leads to bad benchmark conclusions and broken deployments on different hardware.

The fix is to pin and record the exact scheme, not just the bit width. The snippet logs method, bits, group size, symmetry, and which layers were kept in FP16. That metadata is what makes a quantized model reproducible and comparable.

Without it, you can't reliably reproduce a result, can't fairly compare two checkpoints, and can't debug a regression. Treat the quantization recipe like any other critical config: version it, log it, and report it alongside your eval numbers.

The pre-flight checklist ties the whole post together: have you measured quality on a real eval set, calibrated on representative data, chosen bit width to match the model's size, and logged the full method including group size and any precision exceptions? Tick those four and you've avoided every mistake in this post.

The overarching discipline is measure, don't assume. Quantization is a robust, near-lossless technique when done carefully — the failures come from skipping the verification, not from the math.

That closes Day 63. You've gone from the concept of fewer-bit weights, through why memory makes it matter, into the scale-and-zero-point mechanics, a hands-on 4-bit load, and the traps that quietly break quantized models.

Tomorrow we move on to a new topic in the NLP & LLMs track, continuing to assemble the practical toolkit for building and shipping with language models.