Quantization is not the only way to compress an AI model. An article from Machine Learning Mastery presents two other approaches, pruning and knowledge distillation, which reduce the number of values or the size of structures rather than their format. A smaller model can speed up inference, provided it maintains the behavior expected by the application.
Quantization changes how the model’s values are represented. Pruning, on the other hand, sets certain parameters to zero or removes larger structures. Distillation trains a small model, called the ‘student’, to replicate the behavior of a larger model, called the ‘teacher’.
Pruning and Sparsity
The so-called ‘magnitude-based’ method relies on a simple assumption: weights with a low absolute value would have less effect on the output. The provided code example applies this principle to a 1024 × 1024 matrix with a rate of 0.5, meaning half of the weights are targeted for removal.
A pruned matrix is still stored in a dense format. The CSR (compressed sparse row) format only retains non-zero values, along with positional metadata. According to the article, it only reduces memory if this metadata costs less than the removed data, and only speeds up computation if the matrix shape, sparsity level, data type, and hardware have an efficient sparse computation kernel.
Structured Pruning and Distillation
Structured pruning removes entire units, such as attention heads, MLP channels, layers, or blocks. The result can often be expressed as a smaller dense model, easier to accelerate on different hardware. In a transformer’s MLP, each intermediate channel is produced by a row of the up-projection and consumed by a column of the down-projection: removing one necessitates removing the other.
The article specifies that the keep_channels parameter denotes a number of channels, not a fraction. Setting it to 2048 thus reduces an intermediate width from 8192 to 2048 channels. The code ensures tensor shape compatibility but does not prove that the retained channels are the least important: a real method evaluates layer outputs and performs recovery fine-tuning. Removing attention heads requires coordinated modifications, and with GQA, a variant where multiple query heads share a single KV head, arbitrary removal can break these groupings.
Distillation, on the other hand, trains the student from the teacher’s signals: output distribution, intermediate representations, or generated data. The loss relies on a temperature T and the Kullback-Leibler divergence, which is zero when the two distributions coincide. The code example trains two small GPT-2-like models, randomly initialized, for a single step. The article clarifies that it does not produce a useful model: a real experiment requires a pre-trained teacher and representative text.
For a serious comparison, the article recommends measuring, on the same queries and hardware, quality on held-out data, parameter size in bytes, peak memory, prefill latency, and decoding throughput. Pruning or distilling can alter the model’s behavior, even when the number of parameters drops significantly.
