Meta FAIR's Self-Improving Memory Scores 56.0 on Continual

Meta FAIR's Self-Improving Memory Scores 56.0 on Continual Learning Benchmark


Meta FAIR has introduced Self-Improving Memory, a method for AI agents that manage their own memory. The model first discovers its own memory instructions, then trains those skills into its weights. Co-authors include researchers from the University of Washington and Johns Hopkins University.


TL;DR


  • Method: The base model is Kimi-K2.6, a 1T-parameter MoE with 32B active parameters and a 256K context window. Distillation adds a LoRA adapter (rank 64) on attention projections.
  • Runs on: Training hardware is not disclosed. Serving Kimi-K2.6 requires a multi-GPU node; Vast.ai recommends 8x H200. Native INT4 weights are available.
  • Performance: Self-discovered memory prompts average 56.0 across 3 tasks, versus 44.6 for the best human-written prompts.
  • Best result: Codebase, at 64.3 with optimized prompts—a gain of 20.6 over the best hand-written harness.
  • Worst result: Spectrum distillation adds only 2.9 points (46.5 vs. 43.6 for SFT on generic runs).

What Is Self-Improving Memory?


Self-Improving Memory is a two-stage method from Meta FAIR that teaches an agent model to manage file-based memory. The agent works through a stream of subtasks. Each subtask starts with an empty context; only a persistent memory directory, edited with file tools, carries over.


Two prompts control that memory. OFFLOAD updates memory after a subtask. LOAD retrieves notes before the next one. Generic versions leave the strategy to the model. A pilot study in the research post found that no single hand-written strategy wins every...

via MarkTechPost

Related