Chain-of-Models: Cross-Model Auditing for Bias-Robust LLM Judges

Large language models (LLMs) are increasingly deployed as automated judges across a wide range of applications, from content moderation to answer grading. However, their assessments remain vulnerable to cognitive biases—such as sycophancy, bandwagon effects, authority bias, and distraction—that can systematically skew outcomes. While existing mitigation strategies rely primarily on prompt-driven debiasing or human evaluation, these approaches fall short: the former is brittle across different bias types, and the latter fails to scale with the growing demand for automated assessment. To address these limitations, we introduce *Chain-of-Models* (CoM), an automated audit pipeline in which a second model inspects the reasoning trace of the first model before delivering the final judgment. The central design question is whether the auditor should be the same model, a same-family model, or a model from a different family. Through experiments spanning 9 models from 6 families, 4 cognitive biases, and 4 factual datasets, we uncover two key findings about auditor identity. First, standalone bias resistance does not predict audit effectiveness. For instance, Kimi-K2.5 excels as a standalone judge on several biases yet performs poorly as an auditor for Qwen2.5-72B's biased traces, highlighting that strong individual performance does not guarantee effective cross-model oversight. Second, the optimal auditor is bias-specific: GPT-4o proves strongest for bandwagon, authority, and distraction biases, while GLM-5 leads on sycophancy. We operationalize these insights into a per-bias auditor selection rule that, given the bias type, scores candidate auditors across three dimensions: functional diversity, per-bias standalone resistance, and calibrated audit effectiveness. Under a calibration/test split, our selection method achieves the highest accuracy across the four biased slices—0.884 compared to 0.824 for the strongest single fixed auditor and 0.805 for the no-audit baseline. This represents a significant improvement in bias robustness without sacrificing scalability. To support reproducibility and further research, we publicly release our data, configurations, and an LLM-agent skill at the project repository: https://anonymous.4open.science/r/chain-of-models-B585.

via ArXiv CL