CalibratedAgentsWhy LLM-as-a-judge fails in production: the base rate problem
A judge has one dial. You need two.
By Calibrated Agents Research Team · 8 Sep 2026 · Updated 8 Oct 2026 · 5 min read
5 wrong answers in 100 · a judge with 90% recall and 15% false alarms
24%of the judge's flags are real
LLM-as-a-judge has good recall and no precision control. In production, where most answers are correct, that turns into a flood of false positives. Here is the arithmetic, and what a calibrated verifier returns instead.
A fair account of the judge
LLM-as-a-judge is the first thing every team reaches for, and it deserves a fair account. It is just a prompt, nothing to train, cheap to set up, and it can catch many of the failures. In our own testing it has good recall. It is not that a judge can't tell the failures among the data; it can. That is not where the problem is.
NOTE · A fair account: we run judges ourselves, as baselines, with the same fast models we use.
The problem shows up in production. A judge can offer you high recall, but that is the only thing it can offer: high recall at the cost of precision. It can catch all the failures, but it flags many more answers besides. You cannot run a judge that catches 90 of 100 failures and also flags 200 correct answers: your reviewers drown, and after a week nobody reads the flags.
For depth, see how a calibrated judge compares with Ground on RAGTruth++ and LLM-AggreFact.
Why the base rate breaks it
Most answers in production are correct. Say 5 in 100 are wrong. A judge that catches 90% of failures with a 15% false-alarm rate produces, per 100 answers, 4.5 true catches and 14 false flags. Three of every four flags are noise. Precision is 24%, and there is no threshold to turn, because the output is a yes or no from a prompt.
NOTE · Precision = true catches ÷ all flags = 4.5 ÷ (4.5 + 14) ≈ 24%.
Three of every four flags are noise.
More votes, bigger models
Teams respond with self-consistency: ask the judge five times in parallel. Failures reduce, but at five times the cost, and there is still no real operating point: five votes give you thresholds at 0, 20, 40, 60, 80 and 100 percent. To get an operating point with 1% granularity from self-consistency you need 100 parallel calls, and that never happens. The other answer is a frontier reasoning model: better, and nobody in a request path waits 20 seconds for it.
NOTE · Five votes give only six possible thresholds: 0, 20, 40, 60, 80, 100%.
For depth, see how even fast decision models slow down as inputs grow.
What the second dial looks like
A verifier that emits a probability rather than a verdict lets the customer pick the point: 1% false alarms for a compliance queue, higher recall for an internal QA sweep, and know what they bought. That requires two things a prompt doesn't have: a classifier over signals that can be measured, and a labelled corpus of failures to measure it on, held out by source document so the number is honest.
NOTE · Held out by source document: no sentence from a test document is ever seen in training.
We measured it
On RAGTruth++, 408 AI-written answers whose hallucinated passages were marked by two annotators, we gave Ground and Jev used as a judge the same 50 or 100 labelled answers to calibrate on, and scored both on the rest. Ground found the hallucinated sentences 4.0 to 6.2 points more accurately. Calibrated for precision with 100 labels, 75.5% of the sentences it flagged were hallucinated, against 69.5% for the judge, and it still found more of them.
On whole-answer verdicts, a well-calibrated judge holds its own: the two are level on RAGTruth++ and on LLM-AggreFact (79.4 balanced accuracy each, on 1,397 answers). The difference is in the second dial: knowing which sentence is wrong, at the precision you choose.
For depth, see the full results on both benchmarks.
How to use this with CalibratedAgents Ground
Ground returns a calibrated probability that the answer contains something its source does not support, the sentence that's wrong with the passage behind it, and operating points whose precision we measured. Try it on an answer, or read the papers and cases this rests on in the reading list.
Try it on an answer How Ground works API docsFrequently asked questions
Is LLM-as-a-judge accurate?
It catches failures well: in our testing its recall is good. What it lacks is precision control, because a yes or no from a prompt has no threshold to tune.
Why do a judge's flags overwhelm reviewers?
Because most production answers are correct. At 5 failures in 100, a judge with 90% recall and 15% false alarms raises about 14 false flags for every 4.5 real ones.
Does self-consistency fix it?
Partly, at a cost. Five votes reduce failures but cost five times as much and give only six possible thresholds; fine-grained control would need about 100 calls.
What is an operating point?
The threshold at which an answer is flagged, with its measured recall and precision. A calibrated probability lets you choose one, such as 1% false alarms for a compliance queue.
How is Ground different from a judge?
Ground returns a calibrated probability you can set your own threshold on, plus the sentence that's wrong and the passage that proves it. On RAGTruth++, calibrated on the same labelled answers as a judge, it found hallucinated sentences 4 to 6 points more accurately; on whole-answer verdicts the two are level.
Sources
- Wang et al., Self-Consistency Improves Chain of Thought Reasoning in Language Models (2022)
- LLM-AggreFact leaderboard and paper (Tang, Laban and Durrett, 2024)
- TypeSafe, Jev System One documentation
- CalibratedAgents reading list
- Base-rate arithmetic: our own, as worked in the text.
Cite this
CalibratedAgents Research (2026). Why LLM-as-a-judge fails in production: the base rate problem. CalibratedAgents. https://calibratedagents.com/blog/why-not-an-llm-judge