We built a multi-agent pipeline where one agent translated a user's natural-language question into a SQL query, and a second agent—the judge—determined whether that query was safe and accurate enough to execute automatically or if it needed human review before running. For several weeks, the system performed well enough that we stopped monitoring it closely.
Then a query that should have been flagged was approved and executed. Nothing catastrophic happened: no data was lost, no write went to the wrong table, but the returned result was completely wrong. The query had silently dropped a filter clause that the user's question clearly implied. It took us a confused half-hour with an analyst to realize that the number presented was simply incorrect—wrong, yet confidently delivered by a judge that had approved it without hesitation.
That incident made us stop treating "LLM judge approved it" as equivalent to "it is correct." We began treating the judge itself as a component that required its own testing.
What's in this article
- The incident and the first assumption
- What the judge was actually doing
- How a judge lies to you
- The fix that helped
- Calibrating the judge against humans
- Conclusion
The incident and the first assumption
My initial assumption was that the judge had made a one-off mistake. LLMs are not perfectly reliable; a bad call happens, but you move on. So I re-ran the same query and the same judge prompt in isolation, expecting either a repeat of the mistake or a correction. The judge approved the query again, with the same missing filter and the same confidence. This wasn't a random miss—it was a reproducible pattern, indicating something structural was going wrong.
To get to the root of the issue, I pulled a batch of past judge decisions and compared them against what a human reviewer would have said about the same queries. That's when the pattern became specific enough to name.
What the judge was actually doing
The generator agent and the judge agent were built on the same underlying model. This wasn't a deliberate design choice—it was simply the model we had standardized on for cost reasons across the entire pipeline. When I swapped in queries generated by a different model (with the same task and schema, and comparable quality on manual inspection), the judge became noticeably stricter. It caught issues in the other model's queries that it had been letting through in its own model's queries.
That is self-preference bias, a documented behavior in LLM-as-judge research, not unique to our setup. What I hadn't appreciated until it caused an actual incident is how consistent the effect can be. This wasn't a judge that was randomly generous; it was a judge that was specifically generous toward outputs written in a style and structure it recognized as its own.
Here's a stripped-down version of what our judge prompt looked like at the time, simplified for illustration (not the real production prompt):
JUDGE_PROMPT = """
You are reviewing a SQL query generated for the following user question.
Approve it for automatic execution, or flag it for human review.
User question: {question}
Generated SQL: {sql}
Schema: {schema}
Return JSON only:
{{"decision": "approve" | "flag_for_review", "reasoning": str}}
"""
def judge_query(question, sql, schema, client, model="gpt-4o"):
response = client.chat.completions.create(
model=model,
messages=[{"role": "user", "content": JUDGE_PROMPT.format(
question=question, sql=sql, schema=schema
)}],
temperature=0,
response_format={"type": "json_object"},
)
return json.loads(response.choices[0].message.content)
This setup was not inherently bad—the judge worked well for the easy cases. The problem emerged in edge cases where the query had subtle flaws that required careful reasoning to detect. The judge, biased toward its own model's output, often failed to spot these flaws.
How a judge lies to you
Self-preference bias is only one way an LLM judge can mislead you. Another is reasoning inflation: the judge provides a plausible-sounding rationale that is actually incorrect. In our case, the judge would often say something like "The query correctly captures all conditions from the user's request" even when a filter was missing. The reasoning was confident and polished, but factually wrong. This is dangerous because you start trusting the rationale rather than checking the query itself.
We also saw overconfidence in familiar patterns. The judge tended to approve queries that followed common templates it had seen before, even if those templates contained subtle errors. It was not just bias—it was a form of pattern-matching that bypassed critical evaluation.
In 2026, with the rapid adoption of LLM judges in production systems (from code review to content moderation), these failure modes have become more widely documented. Research from groups like Anthropic and OpenAI has confirmed that self-preference bias is not only real but can be amplified when the judge and generator are fine-tuned on the same data. Yet many teams still rely on such judges without adequate guardrails.
The fix that helped
The first fix was to decouple the judge from the generator. We switched the judge to a different model—one that was not used to generate queries. This immediately reduced the self-preference bias. In our tests, the cross-model judge was 15% more likely to flag a problematic query that the same-model judge had approved. It was a simple change with a significant impact.
Second, we added a verifier step that re-ran the query logically: we parsed the natural-language question for all explicit constraints and checked whether the SQL included them. This was a deterministic check (not LLM-based) that caught missing filters, dropped joins, and other structural issues. It was not perfect—it couldn't understand semantic nuances—but it was a reliable safety net for the most common failure modes.
Third, we introduced uncertainty thresholds. If the judge's confidence (as expressed in its reasoning or via a separate confidence score) fell below a set threshold, we automatically escalated to human review. This meant we didn't have to rely on the judge's binary decision alone.
Finally, we started logging all judge decisions with their reasoning, and we periodically reviewed a random sample. This gave us ongoing visibility into judge behavior, allowing us to detect drift or emerging biases.
Calibrating the judge against humans
To make the judge truly reliable, we needed a way to measure its performance against human judgment. We built a calibration set: a collection of 200 queries (mix of safe and problematic) that were labeled by human SQL experts. We then ran the judge on this set and measured its agreement rate with humans, as well as its precision and recall for flagging issues.
Initially, the judge had a precision of 0.85 and recall of 0.70 against the human labels on the calibration set. But when we broke the numbers down by generator model, we saw a stark difference: with the same model, recall dropped to 0.55 (it missed many issues), while with the cross-model, recall was 0.82. This quantified the bias and justified the switch.
We also used the calibration set to set the uncertainty threshold. By analyzing the judge's confidence scores on labeled examples, we found a sweet spot: if we escalated everything with confidence below 0.75, we caught 90% of the issues that humans flagged, without flooding our review queue with false positives.
This calibration process should be ongoing. We now re-run the calibration set quarterly, and whenever we change models or prompts, to ensure the judge remains aligned with human expectations.
Conclusion
An LLM judge is not an oracle. It is a component with biases, blind spots, and failure modes that need to be understood and tested. Our incident was a wake-up call that "the judge approved it" is not a guarantee of correctness. By decoupling judge and generator, adding deterministic checks, using uncertainty thresholds, and calibrating against human reviews, we greatly reduced the risk of similar failures.
In 2026, as LLM judges become more common in production, teams must adopt a mindset of continuous evaluation. Ask yourself: Is my judge biased toward its own output? Are there deterministic checks I can add to reduce reliance on the judge? Do I have a calibration set to measure judge performance? These steps may not eliminate all failures, but they can turn a judge that agrees with itself into one that earns your trust.