The one-sentence version
Quantization stores each parameter in fewer bits. Smaller footprint, faster runs, slightly fuzzier answers — the central trade-off of local AI.
Why Shrink a Model?
Remember the rule from the Parameters page: ~2 GB of memory per billion parameters at full (16-bit) precision. A 70B model needs ~140 GB just to hold its sliders — more than any single workstation GPU. Quantization attacks that number at its root.
Each parameter is just a number, and numbers can be stored with more or less precision. Training uses high precision because every decimal place matters while learning. But at run time, most of those decimal places turn out to be noise. Rounding them off barely changes the answers.
Fewer bits per parameter = smaller download, smaller memory footprint, faster inference — at a small cost in precision.
Bits, in Plain Language
Precision is measured in bits — how many digits of the slider position you keep. More bits = more faithful = bigger.
Full precision (FP16). The training-grade original. Reference quality, reference size: ~2 GB per billion parameters.
Half the size, nearly all the quality. ~1 GB per billion parameters. The quality loss is usually imperceptible in everyday work.
The local-AI sweet spot. ~0.5 GB per billion parameters — a 70B model fits in ~40 GB. Slight quality loss, usually fine for chat and Q&A; this is what most self-hosters run.
Aggressive. Runs on very small hardware, but quality degrades noticeably — answers get shallower, instruction-following slips. Last resort.
Reading the File Names
Quantized models are usually distributed in GGUF format (the format used by llama.cpp, Ollama, and most local tools). A typical name looks like:
llama-3.1-8b-instruct-Q4_K_M.gguf
llama-3.1-8b — family, version, size
instruct — the chat-tuned variant (as opposed to base)
Q4_K_M — 4-bit quantization, K-quants, Medium mix
The Q number is the bits-per-weight. Letters after it grade the quality mix — you'll see a ladder like:
If in doubt: Q4_K_M if it fits, step up to Q5/Q6 when memory allows.
What Compression Costs You
Quantization is compression, and compression is never perfectly free. What to expect:
At 8-bit / good 4-bit: everyday chat, drafting, and document Q&A are practically indistinguishable from full precision.
Where it shows first: intricate logic, rare facts, subtle tone, long chains of reasoning — precision-heavy tasks feel the rounding before casual ones do.
What you gain: smaller models run faster and leave memory free for context — often the net experience is better, because you can afford a bigger model at 4-bit than at 16-bit.
Practical rule: a 4-bit 70B usually beats a 16-bit 8B on hard tasks if both fit. Fit the biggest quality model you can at 4-bit; don't chase 16-bit perfection on small hardware.
Why This Is the Sovereignty Enabler
Quantization is the reason "run real AI on one modest server" is true rather than marketing. It's how frontier-class open models fit on hardware a 40-person firm can buy — which is exactly the claim the Sovereign AI track makes. No datacenter, no GPU cluster: download the right GGUF, run it on your box, and the prompts never leave your network.