Abstract
Machine-learning models can predict ICU mortality with high accuracy, yet feature-attribution methods alone rarely provide the clinical narrative required for bedside adoption. Large language models (LLMs) may bridge this gap, and multi-step agentic pipelines represent a natural extension by separating data interpretation, guideline checking, and final explanation. This revised feasibility study preserves the original comparison between a standalone LLM and a pre-specified agentic pipeline while making the key clinical findings more explicit.
Using the retained local eICU Demo artifact set (2,353 ICU stays; 8.1% mortality), XGBoost achieved an AUROC of 0.855 (95% CI 0.796–0.906) and an AUPRC of 0.332 (95% CI 0.217–0.494). On a stratified 38-case explanation subset, the standalone LLM produced 1 explanation with explicit outcome leakage, whereas the four-step agentic pipeline produced none. Among the 14 cases overlapping with the SHAP review subset, the standalone LLM demonstrated higher SHAP alignment (mean Jaccard 0.171 vs. 0.077) and direction consistency (92.9% vs. 78.6%), while the agentic pipeline achieved higher guideline grounding (0.762 vs. 0.143), value specificity (0.236 vs. 0.143), and slightly higher plausibility (0.700 vs. 0.671).
Clinically, these results suggest that agentic decomposition may improve safety-relevant grounding and patient-specific detail. However, such pipelines should be paired with attribution-based checks before deployment in high-stakes risk explanation contexts.
1. Introduction
Intensive care units (ICUs) generate vast amounts of patient data, and machine-learning models have increasingly been used to predict mortality risk. While these models achieve strong predictive performance, their opacity limits clinical utility. Traditional explainability methods, such as SHAP, provide feature attributions but lack the narrative structure clinicians need to understand a prediction in context.
Large language models offer a promising solution by generating natural-language explanations. With the rapid advancement of LLM capabilities—particularly in 2026, where models are increasingly integrated into clinical workflows—there is growing interest in agentic pipelines that chain multiple reasoning steps.
This study compares two approaches: (1) a standalone LLM that generates an explanation directly, and (2) a pre-specified four-step agentic pipeline that explicitly separates data interpretation, guideline checking, reasoning, and final explanation generation.
2. Methods
2.1 Dataset
We used the eICU Demo Dataset, a publicly available critical care database containing 2,353 ICU stays. The cohort had an 8.1% mortality rate, representing a realistic clinical risk distribution.
2.2 Predictive Model
An XGBoost classifier was trained to predict ICU mortality. The model achieved an AUROC of 0.855 (95% CI 0.796–0.906) and an AUPRC of 0.332 (95% CI 0.217–0.494), providing a solid baseline for explanation generation.
2.3 Explanation Approaches
Standalone LLM
A single LLM prompt was used to generate explanations from patient features and model predictions. This approach leverages the model's intrinsic reasoning abilities without additional structure.
Agentic Pipeline
The pre-specified agentic pipeline followed four sequential steps:
- Data interpretation: Extracting and summarizing key patient features
- Guideline checking: Mapping features to clinical guidelines
- Reasoning: Synthesizing information into a coherent risk assessment
- Explanation generation: Producing the final clinical narrative
This decomposition mirrors how clinicians naturally approach risk assessment, potentially improving transparency and auditability.
3. Results
3.1 Outcome Leakage
Outcome leakage—where explanations inadvertently reference the true outcome rather than model reasoning—was observed in 1 of 38 cases (2.6%) with the standalone LLM. The agentic pipeline produced zero instances of leakage, suggesting that structured decomposition may mitigate this risk.
3.2 Alignment with SHAP Attribution
On the 14 overlapping cases, the standalone LLM achieved higher alignment with SHAP feature attributions (mean Jaccard: 0.171 vs. 0.077) and direction consistency (92.9% vs. 78.6%). This indicates that the standalone approach more faithfully reflects the model's underlying decision process.
3.3 Clinical Quality Metrics
The agentic pipeline excelled on clinically relevant quality metrics:
| Metric | Standalone LLM | Agentic Pipeline |
|------------------------|----------------|------------------|
| Guideline grounding | 0.143 | 0.762 |
| Value specificity | 0.143 | 0.236 |
| Plausibility | 0.671 | 0.700 |
Guideline grounding reflects how well explanations reference established clinical protocols. Value specificity measures the precision of physiological values cited. Both were substantially higher in the agentic approach.
4. Discussion
4.1 Trade-offs in Explanation Design
The results reveal a fundamental trade-off. Standalone LLMs better reflect the model's actual reasoning (higher SHAP alignment), while agentic pipelines produce explanations that are better grounded in clinical guidelines and contain more specific patient details.
This distinction matters: attributions explain the model, while guidelines explain the clinical context. For bedside use, both perspectives are valuable, but they serve different purposes.
4.2 Safety Implications
The absence of outcome leakage in the agentic pipeline is particularly notable from a safety perspective. In high-stakes environments like ICUs, even a single instance of leakage could undermine clinician trust or lead to improper decisions.
4.3 Hybrid Approach
Given these complementary strengths, we recommend a hybrid workflow: use attribution-based checks (e.g., SHAP scores) to verify model fidelity, then refine with agentic decomposition to enhance clinical grounding and specificity. This aligns with emerging 2026 best practices for LLM deployment in healthcare, where reliability and transparency are paramount.
5. Conclusion
This feasibility study demonstrates that both standalone LLMs and pre-specified agentic pipelines can generate useful explanations for ICU mortality predictions. The agentic approach offers superior safety and clinical grounding, while the standalone approach better reflects model reasoning. For high-stakes applications, combining both approaches—with attributional validation as a check on structured clinical narrative generation—appears most promising.
Future work should extend this comparison to larger cohorts, additional ICU outcomes, and prospective clinical evaluation.
Subjects: Artificial Intelligence (cs.AI)
Cite as: arXiv:2608.26109 [cs.AI]
DOI: https://doi.org/10.48550/arXiv.2608.26109
via ArXiv AI