A technical post published on the Hugging Face website describes how to run Qwen3.8-Flash-Next, a 121 GB MoE-type model, on a MacBook with 32 GB of unified memory. By streaming the experts directly from the SSD, the author achieves 13.9 tok/s in generation, about 6% less than Strata, an inference engine specialized for this type of model.
The starting point is personal. The author wants to rely less on online models and run larger models locally. They have a PC equipped with two RTX 4090s, running Windows 11, and an M5 MacBook with 32 GB of unified memory. After an initial attempt with Qwen 3.8 27B, discussions on social media steered them towards Strata and Qwen3.8-Flash-Next. A MoE (mixture of experts) is an architecture where only a portion of the sub-networks, called experts, is activated for each token, i.e., for each processed text fragment. According to the post, the experts represent over 90% of the weights of a MoE model, allowing only the most frequently used ones to be kept in fast memory.
Comparing Offloading Methods
Before choosing Strata, the author compared llama.cpp, vLLM, and Transformers on Qwen3.6-35B-A3B, with a 24 GB A10G card. Strata achieves 82 to 88 tok/s in Q8_0, llama.cpp 63 to 64 tok/s with the –fit on option, and Transformers with Accelerate only 0.16 tok/s. Harry Mellor reran the vLLM tests with better parameters: the fastest configuration, in FP8 with expert offloading, reaches 20.1 tok/s.
The post retains a central idea: transforming the expert cache into a ‘slot pool’, a single preallocated tensor shared by all layers. The expert identifiers provided by the router are redirected to these slots, allowing the use of Transformers’ existing kernel, grouped_mm, without writing specific computation. On OLMoE-1B-7B, the improvement brings the speed to 34.6 tok/s, compared to 38.1 tok/s for classic Transformers.
A 32 GB Mac for Qwen3.8-Flash-Next
For the target model, the experts are quantized to 3 bits, i.e., 49 GB on disk, with a 12 GiB cache on the device. The dense layers are converted to 8 bits, about 5.4 GB, for a perplexity overhead of 0.2% compared to BF16. The n-gram embedding table, 51 GB, remains on disk and is only read when needed. The conversion takes about five hours. After loading, the model occupies about 18 GiB, and a 6,000-token prompt peaks at 25.4 GiB, which, according to the author, is the machine’s limit: they have to close their browser to prevent macOS from starting to use swap.
A 6% Gap with Strata
After several optimizations, measurements on the same M5 and with the same prompts yield 13.9 tok/s in decoding over 200 tokens, compared to 14.8 for Strata. In prefill on a 4,000-token prompt, Transformers reaches 228 tok/s, compared to 220 for Strata. Among the cited improvements are early validation of GPU computations on MPS, which brings about 10%, and the use of Strata’s GGUF weights, which reduces dense layer traffic from 5.4 GB to 3.9 GB per token.
According to the author, the remaining gap is mainly due to back-and-forth between the host and GPU, which accounts for about 10 ms per token. They consider this speed still insufficient for daily use with programming agents: ‘Orthrus continues to do the real work.’ The programming interface remains under discussion with the Transformers maintainers.
