Quantization Without the Math Fear

Smaller numbers, smaller model

A bit is the smallest unit of computer storage: a zero or one. Ordinary model weights may use 16 or 32 bits each. Quantization stores approximations with fewer bits, often 8, 5, or 4. The model then needs less disk space, random-access memory (RAM), and memory traffic, but some answer quality may be lost.

Names such as Q4_K_M label a particular quantization recipe. Q4 roughly means four bits per weight. K_M describes how weights are grouped and which important values get more care. You do not need to decode every suffix; treat it as a build choice that must be tested on your real task, not a promise of quality.

Tip: Benchmark the real task. A small four-bit model that meets an extraction accuracy target is better engineering than a larger model that exhausts the host.