The Judge Agrees With You Exactly Where You Don't Need It

Key Takeaways

• LLM judges reached high agreement with human experts only on questions the judges could correctly answer themselves, when grading without a reference answer.
• That inverts the usual assumption. The judge is most trustworthy on the items you didn’t need it for, and least trustworthy on the ones that motivated the eval.
• Position bias is systematic, not noise: across 15 judges and 150,000+ evaluation instances, it varied significantly by judge and by task rather than behaving like random error.
• Judge quality is not a function of model size. Of 54 models tested for agreement with humans, 27 reached the top tier, and which ones did depended on training strategy, not parameter count.
• The failure mode isn’t a wrong number. It’s a confident number, on exactly the cases where you had no independent way to check it.

There is a specific kind of instrument that is worse than no instrument: one that reads correctly whenever you could have guessed the answer, and drifts whenever you couldn’t.

You would never accept that in a thermometer. It is roughly what the evidence says about using a language model to grade another language model’s work.

What Is LLM-as-a-Judge?

LLM-as-a-judge is the practice of using one language model to score another model’s output in place of a human annotator: handing the judge a response, sometimes a reference answer, and a rubric, and taking back a rating or a preference.

It exists for good reasons. Human annotation is slow and expensive, and most outputs worth caring about are open-ended enough that exact-match scoring cannot grade them at all. If you are evaluating an agent across a whole trajectory rather than a single answer, there is no string to match against. Something has to render a judgment, and hiring people to render thousands of them per deploy is not a real option for most teams.

So the judge is not a shortcut anyone should be embarrassed about. The question is narrower and more practical: under what conditions does its number mean anything?

When Does an LLM Judge Agree With Human Experts?

On questions it could have answered correctly itself. That is close to the whole finding, and it is not a comfortable one.

In 2025, Michael Krumdick, Charles Lovering, Varshini Reddy, Seth Ebner and Chris Tanner built the Business and Finance Fundamentals benchmark: 160 challenging questions with long-form responses authored by financial professionals, who then rated 1,200 model-generated responses. Real domain experts, writing the questions and grading the answers.

Their result, stated plainly in the paper: LLM judges achieved high agreement with the human experts only on questions the judges were able to correctly answer themselves, when grading without a reference answer.

Sit with the shape of that for a moment. It is not “judges are 80% accurate.” It is that the judge’s reliability is conditional on the judge already knowing the answer. Where the model understands the domain, its grades track expert grades. Where it doesn’t, they stop tracking, and nothing in the output distinguishes the two cases. The judge does not return a confidence interval that widens when it is out of its depth. It returns a score, in the same format, with the same apparent authority.

The paper also found the problem largely disappears when the judge is given an expert-written reference answer. That is a genuinely useful result and I’ll come back to it, because it points at the only cheap fix in this literature.

Why Is That Worse Than Having No Eval?

Because the questions a judge cannot grade reliably are the same questions you built the eval to answer.

The reason is which questions you built it for.

Nobody stands up an evaluation harness to confirm that the model can add two numbers. You build it for the ambiguous cases: the ones where reasonable people disagree, where the domain is specialized, where you genuinely cannot eyeball the output and tell. Those are, definitionally, the questions the judge is least likely to be able to answer itself.

So the coverage of the instrument is inversely correlated with your need for it. On the easy band, where you had other ways to check, the judge is reliable and largely redundant. On the hard band, where you have no independent signal, it is least reliable and you have nothing to catch it with.

Running no eval at all leaves you in a known state of ignorance, which is uncomfortable and therefore tends to make people careful. An unvalidated judge replaces that with a number, and a number gets pasted into a dashboard, quoted in a review, and used to justify a ship decision. The failure is not that the score is wrong. It is that the score is confident, precisely where confidence is unearned. That is the same structure as a benchmark whose grader can be satisfied without doing the work: the instrument keeps producing readings after it has stopped measuring, and the readings look exactly like the ones from when it worked.

This is what makes it a liability rather than merely a weak asset. A missing measurement gets treated as missing. A bad measurement gets treated as a measurement.

Can You Just Swap the Order and Average?

Partly. Order effects are real and correctable, but correcting them doesn’t touch the competence problem above.

The order sensitivity is well established. In 2023, Peiyi Wang and colleagues published Large Language Models are not Fair Evaluators, showing that “the quality ranking of candidate responses can be easily hacked by simply altering their order of appearance in the context.” Their demonstration was blunt: with ChatGPT as the evaluator, Vicuna-13B could be made to beat ChatGPT on 66 of 80 tested queries. They proposed three calibrations: generating supporting reasoning before the verdict, aggregating across both orderings, and routing hard cases to humans.

That was an early result, and the obvious response was to assume it would wash out as models improved. It did not. In 2024, Lin Shi, Chiyu Ma, Wenhua Liang, Xingjian Diao, Weicheng Ma and Soroush Vosoughi ran a systematic study across 15 judge models, 22 tasks from MTBench and DevBench, roughly 40 solution-generating models, and more than 150,000 evaluation instances. Their conclusion: position bias “is not due to random chance and varies significantly across judges and tasks.” Prompt length had little influence. What did matter was the quality gap between the two candidates. The closer the call, the larger the bias.

That last detail is the sting. Position bias is smallest when the answer is obvious and largest when the comparison is genuinely close. Once again, the instrument degrades exactly as the question gets harder.

Balanced position calibration does fix the order artefact, and you should use it. But it is worth being clear about what it buys: a judge that is consistently wrong in both orderings is now consistently wrong on average. Averaging two readings from a miscalibrated gauge produces a more stable miscalibration.

Are Some Judges Reliable Enough?

About half the field clears a meaningful bar, but you cannot identify which half by model size.

Steve Han, Gilberto Titericz Junior, Tom Balough and Wenfei Zhou tested 54 models as judges on response-accuracy tasks, measuring agreement with humans through correlation followed by Cohen’s Kappa, an agreement statistic that corrects for the agreement you’d get by chance, which raw percentages do not. Of the 54, 27 reached their top tier: 23 exhibiting human-like judgment, meaning variation within the range of natural human-to-human disagreement, and 4 rated super-consistent, exceeding typical human-to-human agreement. Their models spanned 1B to 405B parameters, open and closed. Their conclusion was that judge excellence depends on training strategy rather than scale.

Half the field clearing a meaningful bar is a genuinely encouraging number, and it deserves saying alongside the rest. But it comes with a corollary: the other half didn’t, and you cannot pick the good ones by reaching for the biggest model you can afford.

Task sensitivity compounds this. The JUDGE-BENCH team, led by Anna Bavaresco and Raffaella Bernardi with a large collaboration, assembled 20 NLP datasets carrying human annotations and tested 11 LLMs on their ability to reproduce them. They found substantial variance across both models and datasets, with reliability depending on the property being evaluated, the expertise level of the human judges, and whether the text being graded was written by a human or a model. Their recommendation is the plain one: validate against human judgments before deploying a judge.

So “is this judge reliable” has no answer. “Is this judge reliable on my task, at my difficulty, against my annotators” has one, and only measurement produces it.

How Do You Validate a Judge Before Trusting It?

Label a sample of your own task by hand, then measure how well the judge reproduces those labels using a chance-corrected statistic, stratified by difficulty, and re-run whenever the judge, prompt or rubric changes.

The literature converges on something unglamorous, which is probably why it appears so rarely in practitioner content.

Label a sample of your own task by hand. Not a public benchmark. Yours, at the difficulty you actually care about. Then measure how well the judge reproduces those labels using a chance-corrected statistic like Cohen’s Kappa, because raw agreement will flatter any judge on a skewed label distribution.

Stratify by difficulty. If you validate on a mixed sample, the easy items will carry the average and hide precisely the failure the Krumdick result predicts. What you want to know is the judge’s agreement on the hard band alone.

Supply reference answers where you can. This is the highest-leverage intervention in the whole literature: the competence gap largely closed when judges were given expert-written references. It converts the task from “know the answer and grade it” to “compare against the answer,” which is a much easier job and one the judge is far better at.

And re-validate when anything moves: the judge model, the prompt, the rubric, the distribution of what you’re grading. A validation is a measurement of a specific configuration, not a certificate the judge carries around.

None of that is exotic. It is the ordinary discipline of not extending belief to a claim before it has earned it, applied to a tool whose output happens to arrive pre-formatted as a number.

Closing Thought

The uncomfortable thing about LLM-as-a-judge is not that it fails. Every instrument fails somewhere, and knowing the failure boundary is what makes an instrument usable.

It is that this one fails quietly, and the boundary runs directly through the region you built it to cover. A judge grading questions it understands is a reasonable labour-saving device. A judge grading questions it does not understand is a random number generator with excellent formatting, and the output of the two is indistinguishable without the human labels that the judge was adopted to avoid collecting.

Which leaves a bill that cannot be avoided, only deferred. You can skip human annotation, or you can know what your eval is worth. There is no configuration that gives you both.

Further Reading

• Krumdick et al., No Free Labels: Limitations of LLM-as-a-Judge Without Human Grounding (2025)
• Wang et al., Large Language Models are not Fair Evaluators (2023)
• Shi et al., Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge (2024)
• Bavaresco et al., LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks (2024)
• Han et al., Judge’s Verdict: A Comprehensive Analysis of LLM Judge Capability Through Human Agreement (2025)

More essays on measurement and what scores establish: AI Evaluation.