Quantization Without the Math Fear

Smaller numbers, smaller model

A bit is the smallest possible unit of computer storage - a single zero or one. An ordinary, unmodified model weight is often stored using 16 or 32 bits. Quantization is the technique of storing an approximation of each weight using fewer bits instead - commonly 8, 5, or 4. Using fewer bits per weight directly shrinks three things at once: the file's size on disk, the random-access memory (RAM) needed to hold it while running, and the amount of memory traffic needed to read those weights during every single inference step. The trade-off is that an approximation is, by definition, slightly less precise than the original - so some amount of answer quality can be lost, though for many practical tasks the loss is small.

You'll see quantized model names like Q4_K_M. Q4 roughly means "about four bits per weight." The K_M part describes a more specific recipe for how the weights are grouped together and which especially important-seeming values get extra care to preserve more precision. You don't need to memorize what every suffix decodes to - what matters is treating the whole name as one specific build choice, to be tested against your actual task, rather than a guaranteed quality tier.

Tip: Benchmark against the real task you actually need done, not a general reputation. A small four-bit-quantized model that reliably hits your specific extraction-accuracy target is better engineering than a larger, higher-precision model that runs out of memory or takes ten times longer on your host.