Your LLM Can Return Perfect JSON and Still Be Wrong

Three weeks after enabling Structured Outputs for a pipeline that parsed payment confirmation messages into transaction records, I noticed that our reconciliation job began flagging a small but steady stream of mismatches. These weren't crashes or malformed rows—just transactions where the amount and sender matched perfectly, but the date was off. Roughly 2 to 3% of a week's volume, enough to notice, but not enough to be immediately obvious.

At first, I assumed it was a timezone bug. It wasn't. When I pulled the raw source messages next to the extracted records, a pattern emerged: every mismatched transaction came from a message that never mentioned a date at all. For example, "Payment received from Chinedu, ₦45,000, ref TXN-82K91"—no date anywhere in the text. Yet the model had filled in transaction_date anyway, almost always using the date the extraction job ran, off by less than an hour.

The schema defined transaction_date as required, and the model couldn't return nothing. So it didn't. I had been treating "the JSON is valid" as the finish line for this pipeline, and for weeks it looked like one. But it isn't. It's the point where a quieter kind of failure becomes possible—one that never throws an error, never fails a type check, and doesn't surface until something downstream depends on the value being real.

Most of what's written about Structured Outputs stops at "it can't return broken JSON anymore," as if that settles the reliability question. It settles one version of it.

The Trap of the Perfect Schema

Structured Outputs solve a real problem. Before native schema enforcement, getting reliable JSON from an LLM meant regex parsers, retry loops, and prompts that essentially begged the model: "ONLY output JSON, no markdown, no preamble." With the modern OpenAI Python SDK and a Pydantic model, most of that category of pain simply goes away:

from pydantic import BaseModel, Field
from openai import OpenAI

client = OpenAI()

class Transaction(BaseModel):
    amount: float
    sender: str
    transaction_date: str  # ISO format

# With structured outputs, the SDK enforces the schema
completion = client.beta.chat.completions.parse(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Payment received from Chinedu, ₦45,000, ref TXN-82K91"}],
    response_format=Transaction,
)
transaction = completion.choices[0].message.parsed
print(transaction)

But the elegance of this setup masks a critical assumption: that the required fields in your schema are actually present in the source data. When they're not, the model doesn't fail—it hallucinates a plausible value to satisfy the schema. In my case, it used the current date, which looked correct at a glance but was semantically wrong.

Why "Valid JSON" Isn't Enough

Valid JSON confirms structure, not semantics. A schema can enforce that transaction_date is a string matching ISO 8601, but it can't verify that the date corresponds to the actual transaction event. When the source message lacks a date, the model has no ground truth to draw from, so it invents one—a behavior that's neither caught by validation nor flagged as an error.

This is a broader issue in LLM pipelines: reliance on schema compliance over data quality. We've become so focused on eliminating syntax errors that we've overlooked the more insidious problem of confident, well-formed, but factually wrong outputs. In 2026, as more enterprises deploy LLMs for high-stakes tasks like financial reconciliation, this distinction is critical. The difference between "valid" and "correct" is the difference between a smooth demo and a production incident.

Consider the downstream impact: a payment reconciliation job that matches transactions might not fail—it will just produce mismatches that require manual review, eroding trust and increasing operational overhead. Over time, this erodes confidence in the entire system.

How to Fix the Quiet Failure Mode

To address this, we need to shift from schema-only validation to semantic validation. Here are some practical approaches:

1. Make fields nullable and explicit: Instead of requiring transaction_date, define it as Optional[str] and instruct the model to return null when the information is missing. This turns an implicit hallucination into an explicit gap that you can handle.


1. Add a "missing fields" mechanism: Include a field like missing_fields: List[str] in the schema, and guide the model to list any required fields that are absent from the source. This gives you visibility into what's incomplete.


1. Cross-check with heuristics: For critical fields, implement a validation layer that checks whether the extracted value is plausible given the source text. For example, if no date is mentioned, flag the transaction for manual review rather than accepting the model's guess.


1. Log and monitor for semantic drift: Track how often the model fills in values for missing fields. A sudden increase could indicate a change in source data patterns or model behavior.


1. Use confidence scores: When available, capture the model's confidence in each field. Low confidence on a required field should trigger an alert.



In my case, switching to a nullable transactiondate and adding a missingfields field cut the mismatch rate dramatically. When the model explicitly returned null for the date, our reconciliation job stopped flagging phantom mismatches and instead flagged transactions needing manual review—a much more useful outcome.




Rethinking Reliability in 2026




As we move further into an era where LLMs handle data extraction and structuring at scale, the definition of "reliable" must evolve. Reliable means more than well-formed JSON; it means outputs that are truthful to the source data, even when that data is incomplete. Structured Outputs are a powerful tool, but they're only as good as the schema design and the validation logic behind them. The moment we mistake valid JSON for correct data is the moment we introduce a class of bugs that are silent, persistent, and costly.




By designing schemas that embrace uncertainty and building validation that checks semantics—not just syntax—we can create pipelines that are truly reliable, not just technically perfect.




---




Benjamin Nweke is a software engineer focused on AI reliability and data pipelines. This article reflects his firsthand experience with LLM-based extraction in production systems.

via Towards Data Science