Quantization and distillation
A model's weights do not need 16 bits each, and a model does not need to be as big as the one that taught it. How floating-point and integer formats trade range for precision, why a handful of enormous activations made large language models hard to quantize, the four methods (LLM.int8(), SmoothQuant, GPTQ, AWQ) that got around them, and how distillation turns a big model's probabilities into a small model's training signal.
55 min
After this lesson you can
- Read the bit layout of fp32, bf16, fp16 and the two fp8 formats, and say what each trades away
- Quantize a tensor with absmax scaling per tensor and per block, and predict what an outlier does
- Explain LLM.int8()'s decomposition, SmoothQuant's rescaling, GPTQ's error feedback and AWQ's scaling
- Distinguish post-training quantization from quantization-aware training
- Write the distillation loss with temperature and explain the T² factor
In this lesson
- Numbers with a floating point
- Squeezing a tensor into integers
- The outlier problem in large language models
- Post-training quantization of the weights: GPTQ and AWQ
- Distillation: a smaller model, taught by a larger one
This lesson is part of Descent Pro
Descent Pro unlocks every lesson on the route, from base camp to the summit. Every plan starts with a free trial.