Stanford's SNAP Lab Open-Sources Memorilla: A 194M-Parameter Latent Memory Module for Frozen LLMs
Stanford researchers have released Memorilla, an open-source memory system that compresses an entire document collection into just 16 memory tokens for a frozen LLM. On FactKG, a frozen Qwen3-8B scored 84.33% with Memorilla, compared to 80.43% when every document was placed in context—achieving that result with 108× fewer document tokens. Code, training recipes, and checkpoints are public under the Apache 2.0 license.
The work comes from Jure Leskovec's SNAP group, with co-authors from the University of Toronto, Penn, and MIT.
TL;DR
- Size: A 194M-parameter trainable memory module that always outputs 16 memory tokens, regardless of collection size. The encoder (Qwen3-Embedding-4B) and decoder (Qwen3-8B) remain frozen.
- Runs on: Linux with CUDA GPUs and Python 3.11. The quickstart requires a single 80 GB GPU (the published recipes ran on an H100 80GB). No quantized variants have been disclosed.
- Performance: Highest reported average across 8 benchmarks (49.68), with top scores on 5 of 8.
- Best result: PersonaMem-v2 accuracy of 44.60%, versus 27.92% for full context.
- Weakest result: PubMedQA accuracy of 59.00%, versus 69.00% for LLMLingua-2.
What Is Memorilla?
Memorilla is a memory module that converts a document collection into 16 vectors for a frozen LLM. A frozen model can only use what fits in its context window, and the two dominant workarounds each carry trade-offs:
- Full-context prompting slows down as history grows.
- Retrieval pastes in the chunks most similar to the question and can miss weaker but relevant clues.
Memorilla takes a third path. Rather than expanding or selectively retrieving from the context window, it distills an entire collection into a compact latent representation that a frozen model can consume directly—keeping inference cost flat while preserving collection-wide signal that similarity-based retrieval may overlook.
Why It Matters in 2026
As agentic systems accumulate long-lived memory and context engineering matures into a first-class discipline, the bottleneck has shifted from raw model capability to how efficiently information can be carried into a frozen model's context. Memorilla's fixed 16-token budget is notable in this landscape: it decouples memory footprint from collection size entirely, a property that matters for always-on agents maintaining persistent state across sessions. Its Apache 2.0 release also makes it straightforward to evaluate against retrieval-augmented and prompt-compression baselines in production pipelines.
Deployment Notes
Memorilla targets Linux with CUDA GPUs and Python 3.11. The quickstart fits on a single 80 GB GPU, consistent with the H100 80GB hardware used for the published training recipes. Teams planning deployments should verify current GPU availability and specifications before provisioning.
via MarkTechPost