RENDER: Controlling Reader-Facing Evidence in LLM Memory Evaluation
Authors: Yuan Si, Simeng Han, Daming Li, Jialu Zhang
Subject: Artificial Intelligence (cs.AI)
Submitted: 5 June 2026 (v1)
Abstract
Memory and retrieval-augmented generation (RAG) evaluations often treat the answering model's input as an implementation detail, overlooking that systems may render the same conversation history as a memory entry, summary, typed record, or raw excerpt. We introduce RENDER, a benchmark control that fixes the conversational context while systematically varying the reader-facing artifact presented to the model.
RENDER combines a five-level packet ladder—which controls when answer-bearing content enters the input—with deterministic templates approximating ChatGPT-style entries, LangChain summaries, MemGPT-style typed records, and raw conversation. Across 500 LongMemEval questions and nine models, matched-budget resolved packets outperform recency-truncated raw dialogue by 42.4–72.6 points. In deployed-style templates, the best-worst spread ranges from 24.6 to 48.8 points per model; under the primary scorer, ChatGPT-style entries achieve higher point estimates than raw conversation on 7 of 9 models. Judge rescoring preserves the positive aggregate effect, though model-specific significance varies. Notably, three models that score 0 percent on formal ledger packets achieve 45.4–53.4 percent on the same facts presented as natural-language entries. The effect persists under retrieval noise and transfers to HotpotQA, indicating that memory and RAG evaluations should report or control the reader-facing artifact.
via ArXiv AI