A book dedicated to the post-training of large language models has just been published. Titled Reinforcement Learning from Human Feedback: Aligning and Post-training LLMs, it is published by Manning and is presented as the fruit of several years of work on open models. Post-training refers to all the steps that follow the initial training of an LLM (large language model) to orient and refine it.
- The book explains post-training of large language models and is freely available online.
- It favors intuition and history over equations, for readers with a computer science background.
- About a quarter of it covers reinforcement learning, from PPO to GSPO and CISPO.
- It also explains distillation, which the author says now shapes practice in leading models.
A book born from a website
The book originates from a website created by the author, who runs the Interconnects blog. He wanted to document key post-training methods for which he found almost no explanation online, including topics like rejection sampling, outcome reward models, or character training. The site, still considered one of the few resources that address these themes in a fundamental and intuitive way, has become relatively popular.
The book favors intuitions and the history of the field over a simple list of equations. It is aimed at a reader who has already completed an undergraduate degree in computer science, not a beginner. Part of its content revisits old blog articles, reworked to connect mathematical concepts. The author specifies that he added a section on on-policy distillation at the last minute and acknowledges that the title is a bit dated.
The text is also freely accessible online. It is accompanied by a 12-hour course, available as slides and video on YouTube, example code with exercises, and model completion comparisons. According to the publisher, a 50% discount is valid on the Manning website until August 19, with the code PBLambert. The book ships from Manning and Amazon US, and from Amazon UK in October.
Reinforcement learning and distillation
About 25% of the book, in word count as well as pages, is dedicated to reinforcement learning (RL). The stated goal is to learn to reason about different algorithms, from the policy gradient theorem to PPO, and then to more recent variants like GSPO and CISPO. The book notably shows how the logic of PPO clipping reduces to six regions based on the sign of the advantage and the value of the policy ratio.
The text also deals with infrastructure. According to the author, the essence of modern RL is a systems problem, involving managing the lag between data and current policy, the gap between training and inference, as well as throughput. The asynchronous architecture, which separates GPUs dedicated to learning and others dedicated to data generation, has remained similar for a few years.
The chapter dedicated to distillation, i.e., using the outputs of one model to train another model, is presented as a way to clarify a practice often described polemically. The book traces the evolution from the 2015 distillation literature to multi-teacher and on-policy distillation, used according to the author by Xiaomi MiMo-V2-Flash and DeepSeek V4.
The author also believes that research should retain the attention of industry players. He indicates that a research paper now takes 3 to 9 months to be integrated into a state-of-the-art model, compared to several years previously in large tech companies. The book aims, he says, to help the reader identify the work that really matters.
