Abstract
Large language models (LLMs) are increasingly employed for title-abstract screening in systematic reviews, yet their decisions often lack calibrated uncertainty. This study demonstrates that an auxiliary BERT+GCN classifier can provide structured uncertainty signals that enhance LLM screening efficiency. We also identify the optimal prompt-delivery strategy to maximize the benefit-to-cost ratio.
We evaluated five LLM prompt-delivery conditions across eight drug-class datasets from the Cohen (2006) benchmark, using 3 seeds × 5-fold stratified cross-validation, yielding 600 fold-level results. A BERT+GCN model trained per fold classifies each test paper as INCLUDE, EXCLUDE, or MAYBE using two spectral tests (algebraic radical and categorical paradox). Experimental conditions varied in information content (none, label, or full scores), selectivity (all papers vs. MAYBE-only), and timing (proactive vs. reactive two-pass). A cross-model pilot against gpt-4.1-mini on three datasets tested cross-generation transfer.
Three key findings emerged:\n1. Full-context delivery produces significant gains in F1 (+0.011, paired Wilcoxon p=0.008) and WSS@95 (+0.050, p=0.039) at a 1.28× token-cost premium, while preserving recall.
- MAYBE-only routing is Pareto-optimal: it achieves the highest mean recall (0.92) and AUC-ROC (0.54) at only 1.05× baseline cost—one-sixth of the full-context overhead.
- The two-pass design escalates 22.2% ± 8.8% of records yet never revises its decision (0% flip rate across all datasets and folds), providing decisive evidence that current instruction-tuned LLMs cannot self-triage.
- A benchmark evaluation of five prompt-delivery conditions across eight drug-class review datasets.
- Evidence that MAYBE-only routing offers the best balance of performance and cost.
- A demonstration that current LLMs cannot effectively self-triage, based on consistent zero-flip rates in two-pass designs.
- A reproducible pipeline with all code, prompts, and cached LLM responses.
- Algebraic radical test: Measures structural irregularity in the feature space.
- Categorical paradox test: Detects contradictory signals between the BERT embeddings and the GCN's graph-based output.
- Baseline: LLM receives no auxiliary information.
- Label-only: LLM receives the auxiliary classifier's predicted label.
- Full scores: LLM receives the auxiliary classifier's full probability scores.
- MAYBE-only proactive: Auxiliary scores are delivered proactively only for MAYBE-flagged papers.
- Two-pass reactive: A first pass screens all papers; a second pass revisits MAYBE-flagged papers.
- Cohen, A. M. (2006). Optimizing feature representation for automated systematic review work identification. AMIA Annual Symposium Proceedings.
- Rahgozar, A., & Mortezaagha, P. (2026). Auxiliary uncertainty signals for LLM-assisted systematic review screening. arXiv:2608.14551.
- [Additional references from the original article would appear here.]
The cross-model pilot shows an identical +0.8% recall uplift for both LLM generations. A per-paper ablation across 20,796 observations demonstrates that the dual paradox test empirically reduces to a one-line logit-gap criterion. We release the full pipeline; the 600-run experiment replays in under one hour from cached LLM responses.
1. Introduction
Systematic reviews are foundational to evidence-based medicine, but their title-abstract screening phase remains labor-intensive and resource-heavy. As of 2026, large language models (LLMs) have become a common tool for automating parts of this workflow. However, LLM outputs in this domain frequently lack calibrated uncertainty, which undermines the trustworthiness and efficiency of the screening process. When an LLM cannot reliably express confidence in its recommendations, reviewers face increased cognitive load and potential errors in study selection.
To address this gap, we investigate whether auxiliary uncertainty signals—generated by a specialized BERT+GCN classifier—can improve the performance of LLM-assisted screening. By integrating these structured signals with LLM outputs, we aim to provide more reliable uncertainty estimates and enable more efficient screening workflows.
This paper contributes the following:
The remainder of this paper is organized as follows: Section 2 reviews related work; Section 3 describes the experimental setup; Section 4 presents results; Section 5 discusses implications; and Section 6 concludes.
2. Related Work
Prior efforts in automating systematic review screening have explored transformer-based classifiers and, more recently, LLM prompting. However, most approaches treat LLMs as standalone classifiers without explicit uncertainty calibration. Some studies have introduced confidence thresholds or ensemble methods, but few have leveraged auxiliary models to inject complementary uncertainty signals into the LLM decision loop. Our work bridges this gap by pairing a lightweight spectral classifier with an LLM under various information-delivery conditions.
3. Methods
3.1 Datasets
We used eight drug-class datasets originally compiled by Cohen (2006), a widely adopted benchmark for systematic review screening. Each dataset contains title-abstract records labeled as relevant or irrelevant to a specific drug class.
3.2 Classification Framework
The auxiliary classifier employed a BERT+GCN architecture, trained per fold. This model classifies each paper into three categories—INCLUDE, EXCLUDE, or MAYBE—using two spectral tests:
3.3 Prompt-Delivery Conditions
We evaluated five conditions:
Each condition was tested across all datasets and folds.
3.4 Evaluation Metrics and Statistical Analysis
We report F1 score, WSS@95 (work saved over sampling at 95% recall), recall, and AUC-ROC. Paired two-sided Wilcoxon signed-rank tests were used to assess significance. All experiments ran on 3 seeds with 5-fold stratified cross-validation, producing 600 fold-level results.
4. Results
4.1 Full-Context Delivery
Delivering full probability scores to the LLM improved F1 by +0.011 (p=0.008) and WSS@95 by +0.050 (p=0.039) relative to baseline, with only a 1.28× increase in token cost. Recall remained unchanged, indicating no adverse effect on sensitivity.
4.2 MAYBE-Only Routing
Routing only MAYBE-flagged papers to the LLM proved Pareto-optimal. This condition achieved the highest mean recall (0.92) and AUC-ROC (0.54) while incurring only a 1.05× baseline cost—one-sixth of the overhead required for full-context delivery.
4.3 Ineffectiveness of Two-Pass Self-Revision
Across all datasets and folds, the two-pass design escalated 22.2% ± 8.8% of records for a second review yet never revised an initial decision (0% flip rate). This finding strongly suggests that current instruction-tuned LLMs cannot self-triage effectively.
4.4 Cross-Model Generalization
A pilot against gpt-4.1-mini on three datasets showed an identical +0.8% recall uplift, indicating that the observed benefits generalize across LLM generations.
4.5 Ablation Study
The per-paper ablation across 20,796 observations revealed that the dual paradox test reduces empirically to a one-line logit-gap criterion, simplifying implementation without loss of fidelity.
5. Discussion
The results offer three actionable insights for LLM-assisted systematic review pipelines. First, providing full-context auxiliary scores is worthwhile when token budget allows. Second, MAYBE-only routing offers the best cost-performance trade-off and should be the default recommendation. Third, system designers should not rely on LLMs to self-correct their own screening decisions; instead, they should build robust triage mechanisms that rely on external signals.
The consistency of the +0.8% recall uplift across two LLM generations suggests that auxiliary uncertainty signals remain valuable as base models evolve. However, the absolute effect size is modest, indicating room for further improvement in how these signals are integrated.
6. Conclusion
We presented a comprehensive benchmark demonstrating that auxiliary uncertainty signals from a BERT+GCN classifier can measurably improve LLM-assisted systematic review screening. MAYBE-only routing emerges as the most cost-effective strategy, while the evidence strongly cautions against expecting LLMs to perform self-revision. As the adoption of LLMs in clinical and research workflows accelerates, these findings provide practical guidance for building reliable, efficient screening systems.
References
via ArXiv CL+LG