Computer Science > Machine Learning

arXiv:2607.20492 (cs) | Submitted on 11 Jun 2026

Authors: Rana Muhammad Usman

Abstract

Language models in production environments rarely write prose; they fill forms—JSON fields, function arguments, extraction templates. In this paper, we demonstrate that the form itself can induce hallucination.

We asked thirteen different models the same question about the same input, varying only the answer format. Inputs were deliberately constructed to be unanswerable: for example, a viral post displaying 12,400 likes but no visible replies, or a support ticket whose call was never transcribed. In free-text responses, GPT-5.5 answered honestly, stating that reply data was unavailable 98% of the time. However, when a JSON field for sentiment was required, the same model fabricated an answer in 40 out of 40 attempts, inventing crowd moods and customer quotes it had never seen.

This pattern persisted forcefully. Required fields drove fabrication rates to 100% in ten of the thirteen models. Even an explicit "insufficient evidence" option only helped frontier models; all nine open-weight models ignored it entirely. A direct instruction not to infer sentiment was overridden by the schema in four out of six models. Importantly, resistance did not scale predictably with model size: within a single model family, the smallest model refused to answer, the mid-sized model fabricated, and the largest model refused again. Honesty under format pressure appears to be a training outcome that current evaluation pipelines are not measuring.

Fabrication is most insidious where hedging is impossible—in required enums, minimum-count arrays, and fields that cannot accommodate disclaimers. We introduce PhantomFill, a benchmark with deterministic scoring and two reportable metrics: the Coerced Fabrication Rate and the Escape Utilization Rate. The fix we propose is a single line of schema modification; the failure we document is pervasive.

Comments: 12 pages, 6 figures. Benchmark and code available at: https://github.com/ranausmanai/phantomfill

Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)

ACM Classes: I.2.7; I.2.6

Cite as: arXiv:2607.20492 [cs.LG] (or arXiv:2607.20492v1 [cs.LG] for this version)

DOI: https://doi.org/10.48550/arXiv.2607.20492

This work highlights a critical and under-measured failure mode in 2026-era language models: coercive schema-induced hallucination. As structured output APIs become the default interface for LLMs in production, ensuring models can safely refuse to answer when evidence is insufficient is an urgent safety and reliability concern.

via ArXiv LG