Prioritize, Don't Decide: Auditing a Deployed Human-in-the-Loop

Prioritize, Don't Decide: What We Learned Auditing a Deployed Human-in-the-Loop CV Sorter


Authors: Hadi Mohammadi, Jeroen Sparla, Alexandros Paliouras, Georg Krempl, Robert A. Bagheri


Subjects: Computation and Language (cs.CL); Information Retrieval (cs.IR); Machine Learning (cs.LG)


Submitted: 17 November 2026 · arXiv: 2611.23359v1 [cs.CL] · Comments: 11 pages, 4 figures, 6 tables. Under review at the EACL 2027 Industry Track




Overview


A single academic vacancy can draw hundreds of applications, and the first screening round consumes the bulk of a committee's time. This paper describes a CV Priority Sorter deployed on a national platform for academic vacancies and integrated with its vacancy database. Rather than making decisions, the system proposes a reading order for a pile of CVs and leaves every judgment to the committee. It pairs a frozen multilingual transformer encoder with a per-session linear model that retrains after each label and is discarded when the session ends.


The approach works: on a pool of 300 synthetic CVs, a committee reading in the tool's proposed order reaches 95% of the relevant candidates after reviewing half the pile, whereas an arbitrary order requires nearly the entire pile. But this paper is about what surfaced when the authors audited that claim—not about the system itself. Three findings ultimately changed the product, and a fourth points toward where to look next.




Key Findings


1. The Evaluation Pool Was Sortable by Document Length Alone


The synthetic pool turned out to be sortable by document length alone, which renders precision-style metrics on it uninformative. Regenerating the pool under a shared length budget removes this shortcut—and the core reading-depth result still holds.


2. Reported Agreement Scores Track Confidence, Not Ranking


The four-level agreement score reported by such systems—including this one—is effectively a binary probability cut at fixed thresholds. As a result, it follows the classifier's confidence rather than its ranking, and it falls as labels accumulate.


3. Uncertainty Sampling Increases the Committee's Total Reading Load


Uncertainty sampling is the obvious way to implement a "suggest the next CV" button. It leaves model quality intact, but it raises the committee's total reading load above the no-label baseline, because the CVs it asks about are precisely the ones that would otherwise be skipped.


4. Stronger Embeddings Help Most at Cold Start


A hosted LLM embedding model lifts zero-label ordinal discrimination from 0.66 to 0.80 under length control, then stops mattering once the session model has roughly two dozen labels. A local cross-encoder matches that cold-start gain without sending data off-site.




Implications for Human-in-the-Loop Design


The audit-driven findings carry practical lessons for deployed screening tools in 2026 and beyond:


  • Validate your benchmarks. Synthetic evaluation pools can encode shortcuts (such as length) that inflate apparent performance. Control for them before trusting precision-style metrics.
  • Report ranking metrics, not just agreement scores. Threshold-based agreement measures conflate confidence with quality and drift as labels accumulate.
  • Weigh annotation cost against model quality. Active learning strategies like uncertainty sampling can improve the model while increasing the human's total effort—an important trade-off as human-in-the-loop hiring tools scale.
  • Match embedding investment to session stage. Hosted LLM embeddings offer strong cold-start gains; local cross-encoders can deliver comparable benefits while preserving data privacy.

For each finding, the authors report the fixes it forced—offering a candid template for teams deploying and auditing high-stakes human-in-the-loop systems.




Citation


Mohammadi, H., Sparla, J., Paliouras, A., Krempl, G., & Bagheri, R. A. (2026).
Prioritize, Don't Decide: What We Learned Auditing a Deployed Human-in-the-Loop CV Sorter.
arXiv:2611.23359 [cs.CL].

via ArXiv CL+LG

Related