Rethinking Next-Visit Diagnosis Prediction Beyond Aggregate Metrics: Retrieval–Memory Multi-Agent Adjudication

Author: Heejoon Koo

arXiv: 2611.03826 [cs.CL]

Submitted: 2 November 2026

Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Information Retrieval (cs.IR)

Comments: 2 figures, 18 tables, 29 pages

Abstract

Next-visit diagnosis prediction forecasts the diagnoses documented at a patient's next encounter, turning retrospective electronic health records (EHR) into prospective clinical prediction. However, conventional aggregate evaluation obscures the multifaceted nature of this task. Distributionally, models must handle variable-size diagnosis sets and long-tailed labels; temporally, they must distinguish persistent, newly recorded, and recurrent diagnoses—and both dimensions vary with the depth of observed history.

The authors introduce an evaluation framework organized along these dimensions and apply it to rule-based heuristics, LLM-based prompting, retrieval, memory, and multi-agent approaches. The analysis reveals that simply repeating the last visit outperforms most LLM-based approaches overall, despite predicting no new or recurrent diagnoses. It also shows that retrieval and memory exhibit distinct profiles, with their relative strengths varying across temporal transitions and backbone models.

Motivated by these findings, the authors propose LONGINUS, a retrieval–memory multi-agent adjudication framework that combines evidence retrieved from patient records with a structured memory of the patient trajectory. Using cross-view agreement as a reliability heuristic, LONGINUS preserves shared predictions and adjudicates disagreements without revealing their source. Across MIMIC-III and MIMIC-IV with four LLM backbones, LONGINUS achieves the best transition-specific performance among LLM-based methods in nearly all settings while remaining competitive on aggregate metrics.

Key Contributions

• A multi-dimensional evaluation framework for next-visit diagnosis prediction that moves beyond aggregate metrics, accounting for diagnosis-set variability, long-tailed label distributions, and temporal transition types (persistent, newly recorded, recurrent).
• A systematic comparison of rule-based heuristics, LLM prompting, retrieval, memory, and multi-agent methods, highlighting the limitations of aggregate evaluation and the distinct strengths of retrieval versus memory approaches.
• LONGINUS, a retrieval–memory multi-agent adjudication framework that integrates retrieved patient evidence with structured trajectory memory and uses cross-view agreement as a reliability heuristic for resolving disagreements.
• Empirical validation on MIMIC-III and MIMIC-IV across four LLM backbones, demonstrating best-in-class transition-specific performance among LLM-based methods in nearly all settings.

Keywords

Next-visit diagnosis prediction; clinical prediction; electronic health records; retrieval-augmented generation; memory-augmented models; multi-agent adjudication; LLM evaluation; MIMIC-III; MIMIC-IV

via ArXiv CL+LG