To specialize a large language model (LLM) for a specific domain, the classic approach is to fine-tune it, i.e., continue its training on a specific dataset. A tutorial compares this method to a lighter solution: keep the model frozen and train only a small classifier on top. On a news article classification task, the first strategy achieves the highest accuracy, but it requires much more time.
- Fine-tuning with QLoRA reaches the highest accuracy, 87.0 %, but takes much longer.
- Frozen embeddings with a random forest reach 82.0 % in far less time.
- Frozen embeddings suit fast training and low computational load, while avoiding overfitting.
- QLoRA fine-tuning suits tasks where closing the accuracy gap or capturing deep domain nuances is critical.
The first strategy relies on LoRA (Low-Rank Adaptation), an algorithm that freezes the model’s pre‑trained weights and adds small trainable adaptation matrices. These matrices typically account for 1 to 2 % of the model’s size. The tutorial uses a quantized variant, called QLoRA, which keeps the model at 4‑bit precision to save memory. The second strategy, known as frozen embeddings, uses the LLM solely as a feature extractor. Each text is transformed into a numeric vector, called an embedding, from the model’s last hidden layer, and these vectors are used to train a classic classifier: a random forest of 100 trees, from the scikit-learn library.
The experiments were run on Google Colab, using a T4 GPU. The model used is Qwen/Qwen2.5-1.5B, and the dataset is ag_news, which contains news articles classified into four categories. To keep the comparison fast, the authors work with a subset of 800 training articles and 200 test articles.
Contrasting results
With the frozen embeddings strategy, vector extraction takes 96.5 seconds, and classifier training only 2.866 seconds. Accuracy reaches 82.0 % on the test set. The tutorial notes that most of the time is spent on feature extraction, with classifier training being almost instantaneous.
The LoRA strategy shows an accuracy of 87.0 %, with training taking 300.3 seconds. The accuracy gain therefore comes at a cost: according to the tutorial, the whole process took three times longer, and this gap could become markedly larger with bigger datasets.
When to favor one over the other
The authors recommend frozen embeddings when a fast training and low computational load are desired, while also avoiding over‑fitting of the base model. Fine‑tuning with QLoRA, on the other hand, is warranted when narrowing the accuracy gap is critical for the target domain, or when the model needs to learn domain‑specific nuances deeply embedded in that field.
The reasoning changes for purely generative tasks, such as text or code generation, where model adaptation—e.g., via fine‑tuning—is generally required to achieve specialized behavior. The tutorial nevertheless notes that frozen embeddings can be a viable alternative for some generative uses in free deployments, when maximum performance is a slightly negotiable criterion.
