COMED: The Missing Middle Between Routing and Collaboration in Multi-LLM Inference

Authors: Norah Alballa, Wenxuan Zhang, Salma Kharrat, Fares Fourati, Zafar Ayyub Qazi, Mohamed Elhoseiny, Marco Canini

Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)

Comments: Accepted at AACL-IJCNLP 2026

Cite as: arXiv:2609.26913 [cs.CL]

DOI: https://doi.org/10.48550/arXiv.2609.26913

Submitted: 22 Sep 2026

Abstract

No single Large Language Model (LLM) is uniformly reliable across queries, motivating multi-model inference systems that either route among models or combine their outputs. However, routing stops after selecting an initial model, while dense collaboration invokes peers on every query. We show that collaboration is non-monotonic: peers can recover failures that no model solves alone, but can also corrupt initially correct answers.

We introduce COMED (Controlled Model Escalation for Multi-LLM Deliberation), a post-anchor controller for selective cross-model collaboration. COMED uses anchor self-consistency, router margin, and a lightweight peer probe to accept confident answers, verify ambiguous cases, and escalate only when collaboration is likely beneficial. We formalize this trade-off with a rescue-harm decomposition, showing that selective collaboration improves when rescued errors outweigh collaboration-induced harms.

Across medical, scientific, and general reasoning benchmarks, COMED improves fixed and routed anchors in all 16 open-weight settings, with gains up to +10.7 percentage points on MedQA while invoking fewer models and using fewer decoded tokens than dense collaboration. On HLE with frontier models, COMED improves GPT-5.5 from 23.1% to 28.1%, outperforming dense collaboration and achieving the best results.

Background: The Multi-LLM Inference Spectrum

As of 2026, no single LLM dominates across all queries. This reality has driven the rapid adoption of multi-model inference systems, which broadly fall into two categories:

Routing: Selects one model per query based on a router's prediction. Efficient, but stops after the initial selection and leaves no mechanism for recovery if the chosen model fails.
Dense collaboration: Invokes multiple peer models on every query and combines their outputs. More robust in principle, but computationally expensive and — as this work demonstrates — not reliably beneficial.

The Non-Monotonicity Problem

The paper's central empirical finding is that cross-model collaboration is non-monotonic:

Rescue effect: Peers can recover failures that no model solves alone, improving overall accuracy.
Harm effect: Peers can also corrupt answers that the anchor model initially got right, degrading accuracy.

This means adding more collaboration does not monotonically improve results. The net benefit depends on whether rescued errors outweigh collaboration-induced harms — a balance the authors formalize as the rescue-harm decomposition.

The COMED Method

COMED occupies the "missing middle" between routing (one model, no recovery) and dense collaboration (all models, every query). It acts as a post-anchor controller for selective escalation, using three signals:

Anchor self-consistency — measures agreement across the anchor model's sampled outputs.
Router margin — captures the routing decision's confidence.
Lightweight peer probe — cheaply assesses whether a peer is likely to help.

Based on these signals, COMED:

Accepts confident answers without further computation.
Verifies ambiguous cases.
Escalates to peer collaboration only when it is likely to be beneficial.

This selective design avoids the unnecessary cost of dense collaboration while retaining its recovery benefits.

Results

Open-Weight Benchmarks

COMED was evaluated across medical, scientific, and general reasoning benchmarks. Key findings:

Improved fixed and routed anchors in all 16 open-weight settings.
Gains of up to +10.7 percentage points on MedQA.
Invoked fewer models and used fewer decoded tokens than dense collaboration.

Frontier Models on HLE

On the Humanity's Last Exam (HLE) benchmark with frontier models:

GPT-5.5 improved from 23.1% to 28.1% with COMED.
COMED outperformed dense collaboration, achieving the best results.

Significance

COMED offers a principled middle ground for multi-LLM inference: it preserves the efficiency of routing while capturing the recovery benefits of collaboration, without paying the full cost — or incurring the harm — of invoking peers on every query. The rescue-harm decomposition provides a formal lens for deciding when cross-model deliberation is worth the expense, and the empirical results across 16 open-weight settings plus frontier-model HLE evaluations suggest the approach generalizes across model scales and domains.

via ArXiv CL+LG

COMED: The Missing Middle Between Routing and Collaboration in