Model quantization is a way to make an AI model smaller and faster by storing its numbers with less precision. Many neural networks use millions or billions of decimal numbers called weights, and those numbers take up memory. Quantization is like rounding very detailed measurements into simpler numbers that are still accurate enough for the job.
This matters because smaller models can run on phones, laptops, web browsers, and low-power devices more easily.
Key Facts
- Quantization reduces the number of bits used to store model values, such as changing 32-bit floats into 8-bit integers.
- Memory needed is approximately number of values × bits per value.
- A 32-bit value uses 4 bytes, while an 8-bit value uses 1 byte.
- Compression ratio = original model size ÷ quantized model size.
- A simple quantization rule is q = round(x / scale) + zero_point.
- Dequantization estimates the original value with x ≈ scale × (q - zero_point).
Vocabulary
- Quantization
- Quantization is the process of representing model numbers with fewer bits to reduce memory use and often speed up computation.
- Weight
- A weight is a learned number inside a neural network that helps determine how strongly one part of the model affects another.
- Precision
- Precision describes how much detail a number can store, such as many decimal places or only a few possible values.
- Bit
- A bit is the smallest unit of digital information and can have a value of 0 or 1.
- Inference
- Inference is the process of using a trained AI model to make a prediction or produce an output.
Common Mistakes to Avoid
- Thinking quantization deletes parts of the neural network, which is wrong because it usually keeps the same structure but stores the numbers with fewer bits.
- Assuming smaller always means better, which is wrong because too much quantization can reduce accuracy if the rounded values lose important detail.
- Confusing training with inference, which is wrong because quantization is often used after training to make prediction faster, although some models are trained with quantization in mind.
- Forgetting to compare bits or bytes correctly, which is wrong because 32-bit values use 4 bytes each while 8-bit values use 1 byte each.
Practice Questions
- 1 A model has 10,000,000 weights stored as 32-bit floats. How many megabytes does it use if 1 byte = 8 bits and 1 MB = 1,000,000 bytes?
- 2 The same model is quantized from 32-bit values to 8-bit values. What is the new size in megabytes, and what is the compression ratio?
- 3 A phone app needs to run an AI model quickly without internet access. Explain why quantization might help, and describe one possible drawback.