Introduction
Despite significant advances in context window sizes, hallucinations in long-context summarization remain a persistent challenge. Long novels—with their dense information, intricate event narratives, and detailed dialogues—provide a more suitable testbed for studying these hallucinations than news articles or research papers. However, existing research lacks a multi-scale benchmark designed specifically for hallucination detection in long-context novel summarization. Furthermore, the relationship between context length and hallucination patterns remains underexplored.
The LongNovel Benchmark
We introduce LongNovel, a multi-scale, bilingual (Chinese and English) benchmark for hallucination detection in long-context novel summarization. The benchmark is built from two complementary sources: 29 Chinese novels spanning 16,000 to 100,000 tokens, and chapter-level data from the BookSum dataset.
Hallucination Taxonomy
To capture the nuances of hallucination in long-context settings, we define eight distinct hallucination types. These categories are designed to cover both factual inaccuracies and coherence-related issues that arise when summarizing extended narratives.
Data Construction and Quality Control
We employ a combination of Multi-Model Arbitration and Entity-Referenced Hallucination Generation to ensure both data authenticity and a balanced distribution of hallucination categories. For the test set specifically, we performed manual revision of the content to guarantee data reliability. This rigorous curation process distinguishes LongNovel from other benchmarks and ensures that the dataset reflects realistic hallucination scenarios.
Experimental Findings
Our extensive experiments demonstrate that LongNovel presents a substantial challenge for current state-of-the-art models. The multi-scale design allows us to observe how hallucination patterns evolve as context length increases, providing valuable insights for future research on scaling context windows.
Availability
We release LongNovel for future research at https://github.com/BDML-lab/LongNovel. We hope this benchmark will stimulate further work on robust summarization systems capable of maintaining factual consistency in long-context scenarios.
via ArXiv CL+LG