| Quant | ~Size (GB) | Fits in RAM? | Fits on GPU? |
|---|
It fits ≠ it's fast. On a modest CPU expect a few tokens/sec; a GPU with
enough VRAM is far faster. Long contexts need memory well beyond the ~15% headroom assumed here.
What do Q8_0, Q4_K_M, Q2_K … actually mean?
Q8_0 — near-lossless- ~half the size of the BF16 master, quality barely distinguishable. Use it when it fits.
Q6_K / Q5_K_M — high quality, smaller- very close to Q8, a good step down when Q8 is a bit too big.
Q4_K_M — the usual sweet spot- best quality-per-GB for most people; the most common local pick.
IQ4_XS — a touch smaller than Q4_K_M- similar quality, slightly less memory (newer "IQ" method).
Q3_K_M — noticeable quality loss- only if Q4 won't fit.
Q2_K — smallest, last resort- real quality loss; use only when nothing else fits.
The number is roughly the bits kept per weight: higher = closer to the
original model, larger file. Quantization is one-way — you shrink the master, you don't get it back.