Skip to main content

AI glossary Models & Architecture

What is quantization in AI models?

← All glossary terms
In short

Reducing the numerical precision of a model's weights or activations, for example to INT8 or INT4, to shrink it and make inference faster and cheaper.

Explained

Quantization is a technique that stores and computes a model's numbers with fewer bits, for example 8-bit or 4-bit integers instead of 16- or 32-bit floating point. The model takes less memory, moves less data and can run faster on GPUs, TPUs, NPUs or even CPUs. The trade-off is some loss of accuracy, which modern methods keep small for many tasks. Quantization is central to deploying large language models at reasonable cost and to running them on smaller hardware, on premises or on devices.

How it works

A model's weights are usually stored as floating-point numbers. Quantization maps them to a smaller set of values, such as integers in a fixed range, plus scaling factors that allow approximate reconstruction.

  • Post-training quantization: applied to an already trained model, often with a small calibration data set. Methods such as GPTQ target large generative models.
  • Quantization-aware training: the model is trained or fine-tuned while simulating lower precision, so it learns to tolerate it.
  • Weight-only vs weights and activations: quantizing only the weights saves memory; quantizing activations too can also speed up computation.

What you gain and what you risk

  • Lower memory use, so a model fits on fewer or smaller GPUs.
  • Faster inference and lower cost per request in many setups.
  • The possibility of running models on premises, at the edge or on devices.
  • Some loss of quality, usually larger at very low precision and on demanding tasks such as reasoning, code or less common languages.

Gains depend on the hardware and the serving software, which must support the chosen format.

When it makes sense

Quantization is worth considering when serving costs or hardware limits are the bottleneck, when data must stay on premises or on a device, or when latency matters. Before using a quantized model in production, evaluate it on your own tasks and compare it with the original: general benchmarks may not reveal the loss on your specific use case.

It also combines with other techniques. QLoRA, for example, fine-tunes adapters on top of a 4-bit quantized model to reduce the memory needed for training.

Frequently asked questions

What is quantization in LLMs?

It is the reduction of the number of bits used to represent the model's weights and sometimes its activations. A quantized LLM needs less memory and compute, so it is cheaper and often faster to run, at the cost of some accuracy.

What is the difference between INT8 and INT4 quantization?

INT8 uses 8-bit integers and INT4 uses 4-bit integers. INT4 halves memory again compared with INT8, but the loss of precision is greater, so quality needs closer checking.

Does quantization reduce accuracy?

Usually a little. With modern methods and moderate precision, such as 8-bit, the difference is often small for many tasks. At lower precision, or on demanding tasks, the loss can be noticeable, so it should be measured on your own use cases.

What is the difference between quantization and distillation?

Quantization keeps the same model and stores its numbers with less precision. Distillation trains a new, smaller model to imitate a larger one. Both reduce cost, and they can be combined.

Decision guides that use this term

Where it shows up at Thinkia

Further reading

Sources

Updated: 2026-10-21

Want to see how this applies to your organisation?

Talk to an AI Expert