PyImageSearch publishes the latest installment of a series devoted to building DeepSeek-V3 \”from scratch\”, this time focused on model training and text generation. Written by Puneet Mangla, the tutorial details the training loop, hyperparameters, and data used for a small-sized model.
- Cette sixième partie conclut la série sur la construction de DeepSeek-V3 à partir de zéro.
- L’auteur utilise l’API Trainer de Hugging Face pour entraîner le modèle sur TinyStories.
- Il retient 10 000 exemples, répartis en 9 000 pour l’entraînement et 1 000 pour l’évaluation.
- Ce petit modèle ne rivalisera pas avec les grands modèles et peut perdre en cohérence.
This sixth part concludes a series that covered, step by step, the configuration and the Rotary Position Embeddings (RoPE), the Multi-head Latent Attention (MLA), the Mixture of Experts (MoE), the Multi-Token Prediction (MTP), and then the assembly of the architecture in PyTorch. The tutorial presents the \”trainer\”, the component that runs the architecture by confronting it with data.
A training relying on Hugging Face
Training a transformer‑type model involves coordinating data loading, optimization, checkpoint management, and evaluation. The author relies on the Trainer API from Hugging Face, with versions transformers==4.57.3, datasets==4.8.5 and accelerate==1.15.0. The model returns a pair (logits, loss), and a custom compute_loss method adapts its output to the format expected by the Trainer.
Several techniques are implemented. gradient accumulation simulates large batches when GPU memory is limited: with a batch of 4 examples and 4 accumulation steps, the effective size reaches 16 on a single device. The learning rate follows a progressive warmup phase, followed by linear decay. Finally, mixed‑precision training (FP16) can speed up computation, although the gain depends on hardware and workload.
Data and configuration
The model is trained on TinyStories, a dataset of simple children’s stories, designed for vocabulary and grammar suited to small language models. The author retains only the first 10 000 examples, split without shuffling into 9 000 for training and 1 000 for evaluation. This second set therefore comes from the selected subset, not from the official validation split of the dataset.
The text is tokenized with the GPT-2 tokenizer, which uses byte‑pair encoding (BPE). Padding positions are excluded from loss calculation using the value -100. The configuration sets a learning rate of 5e-4, 10 warmup steps, a regularization of 0.01 and 2 training epochs. Checkpoints are created every 50 steps, three are kept, and the final selected model is the one with the lowest validation loss.
To test generation, the author uses a temperature of 0.8 and a top‑k sampling of 50, with 150 new tokens. The story presented as an example is explicitly illustrative and not from a recorded run. The tutorial notes that this small model will not compete with large pre‑trained models and that its outputs may repeat, drift, or lose coherence. The full tutorial and its source code are available on the PyImageSearch website.
