Since the beginning of 2026, Anthropic, OpenAI, and the White House have accused several Chinese companies, including DeepSeek and Moonshot, of having ‘distilled’ their models. However, distillation is first and foremost a common training technique that labs apply to their own models.
Why train a small model from a large one?
The most powerful models are also the most expensive to run, as each response requires significant computing power. Distillation involves teaching a small model, called the ‘student,’ what a large model, called the ‘teacher,’ can do.
The idea was formalized in March 2015 by Geoffrey Hinton, future Nobel Prize winner in physics, Oriol Vinyals, and Jeff Dean, then at Google, in a research paper that gave the technique its name. To illustrate the fact that a model assigns a probability to each possible option, the three researchers give the following example: ‘An image of a BMW has only a very low probability of being mistaken for a garbage truck, but this error is still significantly more likely than confusing it with a carrot.’
How does distillation work?
Applied to large language models, distillation generally follows three steps. First, the large model receives a very large volume of queries, chosen according to the targeted skills (coding, mathematics, writing…), and its responses are saved. The student then learns to reproduce them, and is finally tested to measure the gap that still separates it from its teacher.
Since the arrival of reasoning models, the teacher’s chain of thought is particularly coveted, as it exposes the steps leading to an answer. In January 2025, DeepSeek published, alongside its R1 model, six smaller models distilled from it, built on the Qwen (Alibaba) and Llama (Meta) families.
US and Chinese labs clash over distillation
In a post published in February, Anthropic accused DeepSeek, Moonshot, and MiniMax of having distilled Claude on a large scale, with over 16 million exchanges generated via approximately 24,000 fraudulent accounts. In July, Michael Kratsios, White House advisor for science and technology, claimed that Moonshot had distilled Claude Fable to design Kimi K3.
At the end of September, OpenAI attributed to individuals linked to the same company a campaign aimed at extracting reasoning from its models. On October 12, Anthropic stated that its new protections reduced the reasoning data obtained by these campaigns by 90 to 99%, figures based on its own measurements, which no independent audit has yet confirmed.
The grievances concern the conditions under which the technique is used: networks of fraudulent accounts, circumvention of access restrictions, violation of terms of use. Anthropic also invokes safety, as a model distilled in this way is unlikely to retain the safeguards of its teacher. Critics point out that these labs themselves trained their models on content collected without the consent of its authors. In the absence of a legal framework specific to distillation, it is currently the providers’ terms of use that draw the line between legitimate use and copying.
