# Quantization Canonical page: https://flit.fyi/glossary/quantization [Glossary](https://flit.fyi/glossary) / [How it hears you](https://flit.fyi/glossary#recognition) Also: model quantization, 4-bit, 8-bit Published October 8, 2026 Definition Quantization is storing a model’s numbers with fewer bits each, for example 4 or 8 bits where the original used 16 or 32. It makes the model smaller and lighter on memory, usually at a small cost in accuracy. A model is mostly a very long list of numbers, called weights. As trained, each weight is typically a 32-bit or 16-bit floating point value. Quantization rewrites them in a coarser form. Apple’s documentation defines it as reducing the number of bits that represent a number. This is the reason the same model can appear as several downloads of different sizes, with labels such as 4bit, 8bit or Q4. They are one model at different levels of compression. ## How much smaller Size follows the bits almost directly. Halve the bits per weight and the file is about half the size. One published example makes it concrete. For an 8 billion parameter language model, the 16-bit file is about 15 GiB, the 8-bit file about 8 GiB, and a common 4-bit build about 4.6 GiB. The 4-bit build works out at about 4.9 bits per weight, not exactly 4, because some parts are kept at higher precision. ## What you give up Accuracy, a little. Coarser numbers mean the model’s arithmetic is slightly off everywhere, and the errors add up. How much it shows depends on the model and on how far you go. Apple’s guidance is that compressing to 8 bits, and for some models to 6 or 4, can work with only a slight decrease in accuracy. ## What you gain besides size Apple lists the possible gains of compressing a model as a smaller memory footprint, lower latency and lower power use, alone or together. On a laptop, memory is often the one that matters: a model that fits comfortably leaves room for everything else you have open. ## Reading the labels The label tells you the precision of the weights. - fp32 or float32: full precision, as most models are trained. - fp16, bf16 or float16: half precision. Often the form a model is published in. - 8bit or int8: a quarter of full precision. - 4bit or int4: an eighth of full precision. The usual choice for running large models on a laptop. - Q4, Q5, Q8 with letters after them: the same idea in one common file format, where the letters name the method used. ## Quantization and palettization Apple’s tools offer two ways to store weights in fewer bits. Linear quantization maps the range of the original values onto a small range of integers. Palettization, also called weight clustering, groups the weights into clusters and stores each weight as an index into a short lookup table. Apple’s guidance is that palettization typically works best on the Neural Engine, and that 4-bit quantization can work well for models running on a Mac’s graphics chip. In Flit Flit’s formatting model and its Qwen3-ASR voice models are quantized builds, which is how they fit in a Mac’s memory alongside your other apps. [What stays on your Mac](https://flit.fyi/features#privacy) ## Questions people ask. ### What does 4-bit mean on a model? That its weights are stored in about 4 bits each, where the original used 16 or 32. The file is much smaller, and the model is slightly less accurate. ### Does quantization make a model worse? Slightly. It trades some accuracy for a smaller size and lower memory use. Apple’s guidance is that 8 bits, and for some models 6 or 4, costs only a slight decrease in accuracy. ### Is a 4-bit or an 8-bit model better? The 8-bit model is closer to the original, and the 4-bit model is about half its size. Choose 8-bit if you have the memory and 4-bit if you do not. ### Does quantization make a model faster? It can. Apple lists lower latency among the possible gains, along with a smaller memory footprint and lower power use. ### Can any model be quantized? Most can, with tools that convert the published weights. How well it holds up varies from model to model. ### Is quantization the same as compression? It is one kind of model compression. Others include pruning, which removes weights, and palettization, which stores weights as entries in a lookup table. ## Related terms - [MLX](https://flit.fyi/glossary/mlx) - [Core ML](https://flit.fyi/glossary/core-ml) - [Local LLM](https://flit.fyi/glossary/local-llm) - [Whisper large-v3 turbo](https://flit.fyi/glossary/whisper-large-v3-turbo) - [Neural Engine](https://flit.fyi/glossary/neural-engine) ## Your voice was always faster. 7 days free, then $19 USD once. No account, no subscription, nothing uploaded. [Download for Mac](https://flit.fyi/download) [Buy now · $19 USD](https://flit.fyi/pricing#personal) macOS 14 or later · Apple silicon