The site PyImageSearch publishes a tutorial dedicated to building the architecture of DeepSeek-V3 with PyTorch, a deep learning library. The fifth installment of a six-part series, ce tutoriel, authored by Puneet Mangla, assembles the building blocks presented in previous lessons to create a full language model, which converts tokens, the text units processed by the model, into logits, the scores assigned to each possible token.
- The fifth part of a six-part series builds DeepSeek-V3 from scratch in PyTorch.
- Each transformer block combines MLA attention and MoE using pre-norm and residual connections.
- Weight sharing between embeddings and the prediction head saves about 12.9 million parameters.
- MTP heads train the model and are skipped during inference, and the implementation recomputes context without a key-value cache.
Previous lessons covered model configuration, the Rotary Position Embeddings (RoPE), which encode token positions, the Multihead Latent Attention (MLA), an attention mechanism, and the Mixture of Experts (MoE), which distributes computation across multiple specialized sub‑networks. The Multi‑Token Prediction (MTP), finally, enables predicting multiple tokens at once. According to the tutorial, it improves training efficiency and can speed up inference thanks to speculative decoding.
Each transformer block combines MLA and MoE according to a so‑called “pre‑norm” scheme, where normalization occurs before each sub‑layer. Residual connections pass signals directly from one layer to the next, which, the author notes, helps address the vanishing‑gradient problem in deep networks. The full model then sequentially applies token and position embeddings, a dropout layer, the blocks, a final normalization, a vocabulary prediction head and, optionally, MTP heads.
Weight sharing and multi‑token prediction
The tutorial also presents partage des poids between the token embedding matrix and the prediction head. For a vocabulary of 50 259 tokens and an embedding dimension of 256, this technique saves roughly 12.9 million parameters. According to the author, it often improves performance because the input and output spaces remain aligned.
During training, the outputs of the MTP heads are computed for several future horizons, then their losses are combined with the main loss and with an auxiliary MoE loss intended to balance the load among the experts. In the provided code, the average MTP loss is weighted by 0.3 relative to the main loss. The MTP heads are not used during inference, i.e., text generation.
Model size and memory footprint
According to the author, the model has roughly 30 to 35 million parameters spread over six layers, i.e., 2 to 3 million per layer, plus an additional 2 to 3 million parameters per MTP head. At FP32 precision, 32‑bit, the parameters alone occupy 120 to 140 megabytes. Training, with gradients and AdamW optimizer states, requires about 500 to 600 MB, plus activations. For a batch of 4 examples of 1 024 tokens, these weigh 200 to 300 MB, roughly 800 MB in total, a amount the tutorial deems compatible with current graphics processors (GPU).
In inference, FP16 precision cuts memory usage by about half. The implementation does not use a key‑value cache and therefore recomputes the context at each generation step. The next lesson in the series will cover training this model on real data.
