CrisisBench: A Privacy-Preserving Evaluation Suite for Crisis Counseling AI
arXiv:2611.21622 (cs.CL) — Submitted 11 September 2026
Authors: Akshay Swaminathan, Filip Ramazan, Sharang Phadke, Kevina Wang, Ivan Lopez, Shaked Peleg Azzam, Gloria Ye, Chastin Chung, William Wang, Stephanie Stoll, Ivy Pham, Rebecca Hurwitz, Shreya Shah, Samuel Chuang, Divyanjali Verma, Abhay John, Ehsan Adeli, Nigam H. Shah
Background
Crisis helplines are a cornerstone of mental health systems, yet they are frequently understaffed and underfunded. Large language models (LLMs) could help support counselors, but existing benchmarks fail to capture the escalation, risk assessment, and de-escalation behaviors that are central to crisis conversations. At the same time, privacy restrictions sharply limit the use of real transcripts for evaluation.
Methods
The authors introduce CrisisBench, a privacy-preserving suite for evaluating both single- and multi-turn crisis counseling. The suite comprises:
- 1,085 single-turn tasks drawn from 200 de-identified conversations
- 1,185 patient LLMs derived from hotline metadata
Patient models and a 23-item LLM judge were calibrated against expert counselors to ensure alignment with clinical judgment.
Results
The study evaluated five GPT models alongside two strategies: draft-then-edit and retrieval-augmented generation (RAG).
- Single-turn evaluation: All systems underperformed human counselors on excessive empathy and excessive questioning—two response-level failure modes that matter in crisis contexts.
- Rubric reliability: The rubric achieved a human–human Krippendorff's α of 0.51 and an LLM–human α of 0.54 (Pearson r = 0.80).
- Multi-turn evaluation: RAG scored highest overall (0.961), with all systems falling within 0.058 of one another.
- Competency breakdown: Microskills and style/tone were near ceiling, while productivity and safety/risk accounted for most of the variation between systems.
- Realism check: Counselors rated the patient LLMs as realistic (86%) and useful (92%).
Conclusion
CrisisBench enables scalable, privacy-preserving evaluation of crisis counseling systems. Current models perform strongly on core structural competencies but still diverge from expert counselors on important response-level failure modes. The authors release the rubrics, judge prompts, and evaluation tasks to support further research.
via ArXiv CL+LG
