NVIDIA has released Dory 1.0 4B, a small reasoning model from Nemotron Labs whose central layers form a shared block, applicable one, two, three, or four times per token. According to a MarkTechPost tutorial, this operation allows the same file to run at four different computation levels without retraining.
- NVIDIA released Dory 1.0 4B, a small reasoning model whose central block runs one to four times per token.
- A MarkTechPost tutorial runs the model with llama.cpp using a 3.7 GB Q5_K_M GGUF file.
- The GGUF header sets a default of four loops, while the model card recommends two.
- Users must set the loop count manually and pin the llama.cpp commit, since Dory support is still a draft pull request.
Each additional pass through the block, called a loop, consumes more computation per token without adding any parameters. The tutorial uses the llama.cpp engine and a 3.7 GB GGUF format file, in Q5_K_M quantization, meaning it is compressed to reduce its memory size. The llama.cpp version used corresponds to the commit pinned in NVIDIA’s model card, as Dory support is still a draft pull request. The filename differs from the one indicated in the card, which mentions dory-q5_k_m.gguf, so the tutorial recommends checking its size and SHA-256 hash. The downloaded file matches the checksum published by NVIDIA.
An architecture read directly from the file
The GGUF header describes the model’s structure even before loading. Dory has 72 layer entries, distributed across 16 input layers, 40 recurrent layers, and 16 output layers. The recurrent block represents 1.84 billion of the model’s 3.98 billion parameters.
The header also indicates a default loop count of 4, while the model card recommends 2. A server launched without an explicit option therefore runs with the most expensive setting. According to the tutorial, this parameter must always be set manually.
From this data, the tutorial estimates the cost of each loop. A token passes through the recurrent block once per loop, so four loops apply 2.51 times the parameters of a single loop and read 2.28 times the weight bytes, this reading being the determining factor for memory-bound decoding. The KV cache, the memory that retains calculations from previous tokens, varies depending on the chosen mode. In per_loop mode, it goes from 306 MiB for one loop to 816 MiB for four loops, for a context of 4,096 tokens. In last_loop mode, it remains at 306 MiB regardless of the number of loops.
Measurements to verify predictions
The associated notebook then verifies these estimates on a running server. It measures decoding speed for each number of loops, tests whether the last_loop cache alters the model’s predictions, and compares next-token distributions across several settings. It also runs a paired evaluation on 30 questions from the GSM8K test set, a math problem benchmark, measuring accuracy, answer length, and time per solved question.
The tutorial notes that the model card announces performance on other benchmarks, such as SWE-bench or GPQA, which are not reproduced in this context, as well as the 256K token context and NVFP4 pre-training.
The notebook is available on GitHub, in the MarkTechPost repository dedicated to agent projects and tutorials. The authors emphasize two points: explicitly set the loop count, since the GGUF uses four by default, and pin the commit of the llama.cpp fork, as Dory support remains a draft pull request.
