At the heart of training models like Gemini, Llama, or GPT lies a concept borrowed from information theory: cross-entropy. An article published by Machine Learning Mastery shows how this concept, related to measuring uncertainty, explains how large language models (LLMs) learn to predict the next word.
Measuring Surprise
Information entropy was born in 1948, with the work of Claude Shannon. It is used to measure the uncertainty, or “surprise,” contained in a probability distribution. The article uses the example of two dice. The first is perfectly balanced: each roll is unpredictable, and the entropy is high. The second is heavily weighted to land on face 6: the result is rarely surprising, and the entropy is low. The formula used is: H(P) = – Σ P(x) log₂ P(x). It reaches its maximum when all probabilities are equal, i.e., for a uniform distribution like that of the first die. The more a few outcomes stand out from the others, the more the entropy decreases.
When the Model Guesses the Next Word
An LLM generates text word by word, or more precisely token by token, a token being a unit of text that the model manipulates. At each step, its final layer assigns a probability to each element in its vocabulary, which typically numbers several tens of thousands of words. If the model hesitates, the distribution is spread out and the entropy is high. If it is very confident, it assigns a very high probability to a specific word, for example “tatami” in the sentence “the cat sat on the…”, and the entropy is low. This is where cross-entropy comes in, the loss function used to guide training. It compares the distribution predicted by the model to the real distribution, which assigns 100% to the actually observed word and 0% to all others. Minimizing this value amounts to gradually making the model’s confidence converge towards the correct words. The article notes that this process, repeated over millions or billions of examples, allows the model to learn to produce coherent responses.
An Example in Python
The article proposes a lightweight script, which does not load any real LLM but illustrates the operation of cross-entropy. The vocabulary is limited to five words: “apple”, “banana”, “cat”, “dog”, and “elephant”. The target word is “cat”. Three predictions are compared: “Confident & Correct”, which assigns 0.90 to “cat”; “Confident & Wrong”, which gives 0.80 to “apple”; and “Uncertain (Flat)”, which distributes the probability equally among the five words, i.e., 0.20 each. The results are displayed as interactive bar charts using the Plotly library. According to the author, the third prediction corresponds to maximum entropy, as the model is completely uncertain. The second, although very confident, is wrong, which results in high cross-entropy. The article reminds us that these two situations, uncertainty and error, are handled simultaneously when seeking to reduce cross-entropy during training.
