Your JSON Is Valid but Your Data Is Wrong: Five Failure Modes LLM Structured Outputs Won't Catch
Large Language Models | September 1, 2026 | 9 min read | By Mostafa Ibrahim
Constrained decoding solved a real problem. Before grammar-based methods like Outlines and SGLang, getting valid JSON from a language model was a retry loop: prompt, parse, catch the trailing comma, reprompt. Constrained decoding ended that by forcing token selection through a finite-state machine, ensuring every output parses. Teams adopted it quickly, and schema compliance hit near 100%. But then a quiet assumption crept into production codebases: if the JSON validates against the schema, the data is correct.
BAML's benchmarks say otherwise. On function-calling tasks, unconstrained generation with post-hoc parsing reached 93.63% accuracy, while constrained decoding on the same model scored 91.37%. The always-valid JSON was less accurate than the sometimes-broken JSON.
I started tracking this after a classification pipeline I built began returning plausible but fabricated values on roughly one in twelve runs. The JSON always parsed; Pydantic never complained. It took weeks to notice because every downstream check was structural.
Five failure modes keep showing up. They all produce schema-valid output that breaks your pipeline silently:
- Enum hallucination: valid enum values with wrong meanings.
- Confident fabrication: plausible values in free-text fields that don't match reality.
- Cross-field contradiction: fields valid individually but impossible together.
- Distributional collapse: convergence on safe defaults, losing diversity.
- Array hallucination: fabricated array entries instead of empty arrays.
- Implement semantic checks: Use rule-based systems or secondary LLM calls to verify that structured content matches the source context.
- Monitor distributions: Track output diversity over time to detect collapse early.
- Use hybrid approaches: For critical fields, combine constrained decoding with post-hoc confidence scoring (e.g., using token log probabilities).
- Test with adversarial examples: Include edge cases in your evaluation set to measure robustness, not just schema compliance.
Constrained Decoding: The Problem It Actually Solved
Structured output used to mean hoping the model behaved. Constrained decoding was built to fix that, and it did—just not the whole problem. The progression was real: from prompt-and-pray JSON, where you appended"valid JSON only" to the prompt and rewrote the prompt when the parser failed, to grammar-constrained decoding that made malformed output structurally impossible. Tools like Outlines and SGLang implement this with context-free grammars that restrict token sampling to only those that can lead to a valid parse. The result: 100% schema compliance, a massive win for reliability.
But schema compliance is not semantic correctness. A JSON document can be perfectly valid and still contain lies. The finite-state machine guarantees syntax, not meaning. The five failure modes below stem from this gap, which no validator—Pydantic, Zod, or otherwise—can close because they validate structure, not truth.
Failure Mode 1: Enum Hallucination
Enums look robust: the model can only pick from a predefined set. But the set is your domain knowledge, not the model's. When the model picks an enum value that is syntactically correct but semantically wrong, you get a flawless-looking output that misrepresents intent.
Example: In a sentiment classification task with options positive, neutral, negative, the model might output neutral for a clearly positive review because it's a safer choice or because context was ambiguous. The schema passes; the data is wrong. Fixes: use few-shot examples that include edge cases, add confidence scoring for model assertions, and implement post-hoc semantic checks where possible.
Failure Mode 2: Confident Fabrication
Free-text fields are the wild west. When the model generates a plausible-sounding value that has no basis in the input, you get fabrication. Constrained decoding doesn't help because grammar constraints don't verify content.
Example: In an extraction task, the model outputs a name that doesn't appear in the source document. The JSON parses, the field is a string, everything looks fine. Detection requires cross-referencing against the original input—comparing extracted values with source tokens—which can be automated with simple set intersection or embedding similarity checks.
Failure Mode 3: Cross-Field Contradiction
Each field validates individually, but together they create an impossible combination. This is common in structured data where fields are semantically dependent.
Example: A JSON object for a person contains age: 25 and birthdate: "2000-01-01"—contradictory by 25 years. Pydantic won't catch it unless you add custom validators. Fix: implement cross-field validation rules using logical constraints (e.g., age must match birthdate year) and use model-level constraints where feasible.
Failure Mode 4: Distributional Collapse
Models trend toward safe, common outputs when constrained. Over time, structured outputs lose diversity, converging on a narrow set of values that are almost always valid but rarely accurate for edge cases.
Example: In a categorization task, the model defaults to miscellaneous for anything slightly unusual, even when a specific category exists. The result: high schema compliance but low utility. Mitigation: monitor output distribution across runs, use temperature adjustments, and include diverse examples in prompts to encourage exploration.
Failure Mode 5: Array Hallucination
When a list is empty in the ground truth, the model often fabricates entries to fill it. Constrained decoding only checks that the array contains valid items, not that those items exist.
Example: For a query asking for irrelevant results, the model returns an array with one or more fake but plausible items. This is dangerous for analytics or search pipelines. Detection: cross-validate against input context, and train/evaluate with explicit empty-array examples to reduce false positives.
What This Means for Your Pipeline
These failure modes are not hypothetical. They occur in production with top-tier models. The common thread is that they all produce schema-valid JSON that passes validators, making them invisible to typical QA. The solution isn't to abandon structured outputs—they're a massive improvement—but to add an additional layer of semantic validation.
Practical recommendations:
The Bottom Line
Constrained decoding and structured outputs are essential tools, but they solve syntax, not semantics. Your JSON is valid, but your data might still be wrong. By understanding these five failure modes, you can build guardrails that catch what validators miss—ensuring your pipeline doesn't just parse JSON, but trusts the meaning behind it.
Image by author