Speculative decoding is a technique that speeds up text generation by large language models, without modifying the model or degrading response quality. Presented as capable of accelerating inference by 2 to 3 times, it relies on a smarter generation strategy rather than an architectural change.
- Speculative decoding accelerates LLM text generation by 2 to 3 times, without changing the model.
- A small draft model proposes several tokens, and the target model verifies them in one parallel pass.
- The verification keeps the output statistically identical to what the target model would produce alone.
- Gains depend on the acceptance rate, and they are highest for predictable tasks like summarization and code completion.
The memory bottleneck
Each time a user sends a message to a chatbot, the model generates its response token by token, i.e., fragment of text after fragment of text. This operation becomes an obstacle as models grow, especially for programming assistants, customer‑service bots or real‑time agents that require near‑instantaneous replies.
The bottleneck does not mainly come from compute power. GPUs like the NVIDIA H100 are very fast for matrix calculations. The issue lies in the fact that the billions of parameters of the model, stored in memory, must be transferred to the processor before generating each token. The compute units then remain largely idle. Engineers talk about a system limited by memory bandwidth: adding compute power therefore does not significantly speed things up.
Propose, then verify
The principle is to generate several candidate tokens in one go, then verify them in parallel. The system relies on two models. A small model called draft (draft model) quickly produces a sequence of candidates, usually four to eight tokens. A target model, larger and higher‑quality, evaluates the whole set in a single pass. Since GPUs are designed for parallel computation, processing a batch of tokens takes hardly more time than processing a single token.
For each candidate, the target model compares its own probability to that assigned by the draft. If the target model is at least as confident, the token is accepted. Otherwise, it is accepted with a probability that decreases as the gap widens. A rejected token is replaced by a token drawn from an adjusted distribution, then the loop restarts. According to the presentation of the method, this statistical procedure guarantees that the output follows exactly the same probability distribution as if the target model had generated each token alone.
In the best case, all candidates are accepted and the target model adds an extra token, i.e., five to nine tokens for the cost of one pass of the target model. In the worst case, the first candidate is rejected, which adds a slight overhead, since the draft generation time is still incurred. A mismatched pair of models can therefore slightly slow down generation.
The real gain depends mainly on the acceptance rate, that is, how often the draft’s candidates pass verification. Around 70 to 80 %, a level considered typical for well‑matched pairs on common tasks, the acceleration lies between 2× and 3×. Predictable or repetitive texts produce higher rates and larger gains.
Variants and practical implementation
The original scheme requires a separate draft model, which adds memory and engineering complexity. Several newer techniques dispense with it. Some give up the strict statistical guarantee for a simpler implementation or higher throughput, with, according to the source, generally negligible impact on output quality.
Medusa adds to the target model several additional prediction heads on its final layer, each forecasting a different future position. Eagle, for “Extrapolation Algorithm for Greater Language-model Efficiency”, relies on the model’s internal states rather than its output tokens, which, according to the source, promotes higher acceptance rates on complex or specialized texts. Prompt Lookup Decoding uses no secondary model: it identifies in the prompt and conversation history expressions likely to reappear in the response, and employs them as candidates, which is especially suitable for code editing and document summarization.
Speculative decoding is offered as an option in several inference frameworks, including Hugging Face’s text-generation-inference, vLLM and llama.cpp. According to the source, enabling it most often involves a configuration change.
The technique especially benefits long generations, when the model processes a request at a time and the prompt and response share vocabulary or structure. Question‑answering, document summarization and code completion are among the cases where gains are greatest. In high‑throughput batch service, however, it can reduce throughput, as draft tokens consume memory and compute that could be used for other requests.
