Stanford researchers have released Memorilla, an open-source memory system for large language models (LLMs). The device reads a complete collection of documents and passes only 16 tokens of memory to the frozen LLM. According to the article, this approach achieves better results than an LLM receiving the entire set of documents in its context, while using far fewer tokens.
A compact memory module
An LLM can only use what is in its context window, meaning the text it receives when generating a response. Memorilla offers a third way between full context and retrieval by chunks, which involves injecting a learned representation of the entire corpus into the model. The module contains 194 M trainable parameters and always produces 16 tokens of memory, regardless of the collection size.
The operation takes place in three steps. A frozen encoder, Qwen3-Embedding-4B, transforms each document once, so adding new documents does not require re-encoding the old ones. For each question, 16 learned slots query the entire store using two layers of 8-head cross-attention. Finally, these slots are projected into the embedding space of the decoder, Qwen3-8B, and placed before the question. Only the memory module is trained, in three phases.
Results announced by the team
On the FactKG benchmark, the frozen Qwen3-8B reaches 84.33 % with Memorilla, compared to 80.43 % when each document is placed in the context, with 108 times fewer document tokens. The researchers report an average of 49.68 across eight benchmarks, the highest among the compared methods, and a better score on five of these eight tests. On PersonaMem-v2, accuracy reaches 44.60 %, compared to 27.92 % with full context. These scores are self-reported by the research team.
The article also notes limitations. On PubMedQA, Memorilla scores 59.00 %, while LLMLingua-2 reaches 69.00 %. The summaries in this dataset total only 324 tokens, leaving little room for compression. The system is also described as weaker on fine-grained facts.
Open usage, with hardware requirements
The Memorilla code, training recipes, and checkpoints are released under the Apache 2.0 license. Running it requires Linux, CUDA GPUs, and Python 3.11. The quick start requires an 80 GB GPU, and the recipes were run on H100 80 GB GPUs. The first launch downloads about 25 GB of models. According to the article, response time is 0.74 s for 512 documents on an H100, compared to 0.75 s to 19.9 s for full context, a difference the authors quantify as 27 times. The researchers have also released PersonalizationV4, a dataset of 149 synthetic users with 200 conversations each, under the CC BY 4.0 license.
