Weight-only quantization lightens AI models

Weight-only quantization lightens AI models
3 min read

Weight-only quantization involves storing certain parameters of a transformer model as integers, while keeping the activations in FP16 or BF16. This technique aims to reduce the memory footprint and the cost of reading the weights, which is significant during text generation. A recently published tutorial details its mathematical principle, then the tools to apply it with PyTorch and other inference engines.

During generation, the model reads its weights for each new token, i.e., each unit of text it produces. With a small batch size, moving these weights can take longer than the computation itself. A weight in INT8 occupies half the space of a weight in FP16 or BF16. Quantization can also speed up computation, as many GPUs dedicated to AI perform matrix operations in low precision faster than in FP16. The actual gain, however, depends on the hardware, the compute kernels, and the shape of the models.

The principle of scale and zero point

The presented method relies on an asymmetric affine mapping, which uses a scale and a zero point. The scale sets the gap between two representable values, while the zero point is the integer code corresponding to the real zero. Including zero in the value range allows it to be represented exactly. The tutorial specifies that this example illustrates the calculation and not efficient storage: a uint8 tensor occupies one byte per element, even for less than eight bits. Per-channel or per-group quantizations often better preserve the quality of a large language model’s weights, as each channel or group has its own range.

With PyTorch, the quantize_() function from the TorchAO library replaces the weights of torch.nn.Linear modules in place. Applied to an already trained model, it falls under post-training quantization, without an additional training step. The tutorial shows an asymmetric per-channel INT8 configuration, with activations remaining in BF16. TorchAO also offers hardware-adapted presets, like Int4WeightOnlyConfig, and the transformers library can apply such a configuration when loading a model. The author recommends performing performance measurements on the entire model, as a mathematically valid configuration is not necessarily the fastest.

Gptq, awq and gguf

GPTQ and AWQ methods use calibration data to reduce the effect of quantization errors on the model’s outputs. GPTQ quantizes weights layer by layer and adjusts the remaining weights to compensate for previous errors. The transformers library now relies on GPT-QModel as the GPTQ engine. AWQ, for Activation-aware Weight Quantization, identifies important weights based on representative activations, then seeks scale factors to protect them before quantization to low precision. For this method, the tutorial uses the reference implementation llm-awq, for which one must follow its own specific installation instructions, and the model TinyLlama/TinyLlama-1.1B-Chat-v1.0. The parameter search relies on 128 samples from the validation partition of mit-han-lab/pile-val-backup, trained in sequences of 512 tokens.

The GGUF format, used in the llama.cpp ecosystem, serves as a container for metadata, the tokenizer, and quantized tensors, but it is not a quantization algorithm in itself. The flow proceeds in two steps: conversion of the original model to high-precision GGUF, in bfloat16, then quantization with the Q4_K_M type. This type is mixed, as it uses different encodings depending on the tensors. Its file size therefore does not correspond exactly to four bits per parameter. The tutorial reminds that one must quantize from the best available source, as requantification accumulates approximation errors.

According to the tutorial, evaluation should be performed on representative inputs, using perplexity, accuracy on target tasks, and side-by-side comparison of generations. The provided script measures, on a CUDA GPU, the peak allocated memory and the median generation time of the FP16 model and the GPTQ version of facebook/opt-125m. The tutorial concludes that quantization is only useful if the format, the numerical scheme, the layout of the compressed data, and the inference kernel work together.

S’abonner
Notification pour
guest

0 Commentaires
Le plus ancien
Le plus récent Le plus populaire
0
Nous aimerions avoir votre avis, veuillez laisser un commentaire.x