Activation and Floating-Point Quantization : The Basics to Know

Activation and Floating-Point Quantization : The Basics to Know

Quantizing activations—the values calculated during a language model’s inference—proves more delicate than quantizing weights, which are fixed after training. According to the Machine Learning Mastery article, the SmoothQuant method and the FP8 format offer two responses to this problem, and the LLM Compressor library, from the vLLM project, allows them to be applied.

Why Activations Are Difficult to Quantize

A network’s weights are fixed once training is complete: they can be quantized once, before deployment. Activations, on the other hand, depend on the token being processed, its position in the sequence, and the preceding layers. Their value ranges can change from one query to the next, and some channels may contain persistent extreme values.

Two approaches coexist. Static quantization uses scales calculated during a calibration phase, which is fast, but an input outside the calibrated range may be truncated. Dynamic quantization measures each tensor at runtime: it adapts to each input but adds a computational cost. The article notably presents per-token dynamic quantization, in signed symmetric INT8, where each vector in the final dimension receives its own scale.

Choosing the range also involves a trade-off. Covering the full range of values makes the quantization step coarse for common values, while truncation improves their precision at the expense of extreme values. The best option depends on how the following weights use this vector.

SmoothQuant and the W8A8 Format

The acronym W8A8 refers to weights and activations in INT8 for matrix multiplication. When hardware handles these integer calculations efficiently, this configuration can reduce memory transfers and increase computational throughput. Biases, normalization, or residuals can remain in higher precision.

SmoothQuant handles extreme values by shifting part of the difficulty from activations to weights. The principle relies on a mathematically equivalent transformation before quantization: important activation channels are divided by larger scales, while the corresponding weight columns are multiplied by the same scales. The article specifies that this transformation only demonstrates an exact equivalence; the actual implementation derives the scales from the magnitudes of the calibration weights and activations.

The author indicates that a W8A8 configuration is not guaranteed to be faster than a W4A16 or BF16 format. W8A8 can leverage integer matrix computation, particularly during prefill and for large batches, whereas W4A16 reads fewer weight bytes and may be suitable for small-batch decoding. The article recommends measuring both phases.

Floating-Point Quantization

FP8 does not simply involve removing bits from an FP32 value. In a scaled FP8 scheme, the tensor is divided by a scale, then each result is rounded to a representable FP8 value. Unlike INT8, FP8 values are not evenly spaced: they are finer near zero and more spaced out for large values.

The article notably distinguishes E4M3, with four exponent bits and three mantissa bits, and E5M2, with five exponent bits and two mantissa bits. E4M3 offers more precision, while E5M2 offers a wider range. The choice depends on the hardware and inference engine.

An LLM Compressor recipe called FP8_DYNAMIC converts a model to the Hugging Face format, quantizing weights by nearest rounding and calculating activation scales during inference. A variant combining SmoothQuant and FP8 is also offered. According to the article, current versions of vLLM support FP8 on NVIDIA Ada Lovelace, Hopper, and Blackwell GPUs, as well as on some AMD accelerators, and it recommends checking the compatibility matrix for the target hardware.

S’abonner
Notification pour
guest

0 Commentaires
Le plus ancien
Le plus récent Le plus populaire
0
Nous aimerions avoir votre avis, veuillez laisser un commentaire.x