/
Published on 10.03.2026
TLDR: If you are using an LLM as a judge to evaluate your AI application outputs, you need to validate that judge itself. Unvalidated evaluators create false confidence, letting broken outputs through while you stare at green dashboards. The fix involves treating your judge as a binary classifier, measuring it against expert labels, and iterating until alignment is real.
So you have set up an LLM judge. It is running evaluations, spitting out Pass or Fail verdicts, and the numbers look reasonable. Time to ship, right? Not so fast.
Here is a story that should make you uncomfortable. While building a writer agent called Brown, the team set up an LLM judge to verify generated articles against a golden dataset. They wanted to check structure, idea flow, and content quality. The scores came back looking reasonable. Then they actually compared the judge's verdicts against manual review. The judge was scoring zero when an article used bullet points instead of H3 headers, which was perfectly acceptable. It penalized creativity when transition phrases differed from few-shot examples. And it scored a perfect one when paragraphs did not flow into each other at all, completely missing a real quality issue. The evaluator was broken, not the agent.
This is not an isolated case. Anthropic reports seeing eval scores jump from 42% to 95% after fixing grading bugs and ambiguous task specifications. The agent was fine all along. Let that sink in: unvalidated evals are worse than no evals at all. You get false confidence, push broken outputs because the numbers say they are good, and hear about problems from frustrated users instead of your test suite. The 10 to 20 percent of incorrect signals hide silently and contaminate every decision built on those scores.
The core insight is straightforward: your evaluator is another AI model making binary predictions. It needs a test set, metrics, and mapped failure modes like any other model. LLM judges are inherently non-deterministic. They hallucinate, carry biases, and drift. Alignment with human evaluators varies wildly by task, with some teams achieving high agreement after careful iteration while others struggle to break 70 percent on subjective criteria.
The first step is partitioning your labeled data correctly. You cannot build and validate on the same examples because that is grading your own exam. Split your dataset into three sets: train at 60 percent, dev at 20 percent, and test at 20 percent. The train set provides your few-shot examples and rubric foundation. The dev set is your iteration sandbox where you run the judge, find disagreements, adjust the prompt, and repeat. The test set stays locked until you are done iterating, giving you an unbiased final score on data the evaluator has never seen.
In practice, 100 labeled examples mean 60 powering the prompt, 20 for tuning, and 20 for the final honest check. Below 50 labeled examples, your numbers become too noisy to act on. And watch out for class imbalance: if 90 percent of your traces are Pass and only 10 percent are Fail, you need to balance the classes before those metrics mean anything.
Since your judge outputs Pass or Fail, you are dealing with a standard confusion matrix. True Positive means both judge and expert agree the output is good. True Negative means both agree it is bad. False Positive means the judge lets a bad output through. False Negative means the judge is overly harsh on a good output.
Here is where teams get burned: accuracy alone is dangerously misleading. If your agent generates 80 articles, 70 are correct, and 10 are broken, a judge that labels every single one as Pass achieves 87.5 percent accuracy. Looks great on a dashboard. Catches zero failures. Completely useless.
You need precision, recall, and F1. Precision tells you how trustworthy the Pass verdicts are. Recall tells you how many actual Passes the judge finds. F1 gives you the harmonic mean. But critically, always check these metrics on the minority class. A judge with 73 percent overall accuracy might have only 20 percent recall on failures, meaning it misses 80 percent of structural problems. If those minority-class numbers are low, enrich your few-shot prompts with more failure examples, particularly the subtle cases.
What is missing from this discussion is the cost dimension of getting this wrong in production. The article focuses on metrics and iteration loops, but does not quantify what happens when those 16 false positives out of 80 reach actual users. In a production system serving thousands of requests, that 20 percent miss rate on failures compounds into a significant trust problem that no amount of post-hoc metric improvement can undo.
The refinement workflow is systematic: start with 10 to 20 few-shot examples from the train set, run against the dev set, compute precision, recall, and F1, then identify every disagreement. Expand few-shot examples by incorporating disagreements that reveal real patterns, re-run, and re-measure until dev set alignment hits your target threshold.
A practical consideration the article raises but does not emphasize enough: few-shot examples translate to input tokens, which translate to cost. You want your few-shot set as minimal yet diverse as possible. There is a real tension here between evaluation quality and operational cost that teams need to navigate deliberately.
Expect at least 3 rounds of iteration. If you are still far below target after 10 iterations, the task may require human judgment that no prompt can replicate. And here is an important nuance: read the LLM judge critiques, not just the metrics. Critiques tell you whether the judge was wrong or whether the expert actually missed something. Sometimes the expert labels need updating, not the judge.
Randomness comes from two directions: the judge produces different scores on the same input, and the agent takes different paths each run. The article recommends several stabilization strategies. Scale the dataset, because larger datasets smooth out noise. Aim for at least 50 samples per class. Use the strongest available model for your judge, since it should be at least as capable as the system it evaluates. Structure prompts with chain-of-thought so the judge walks through each criterion before delivering a verdict. And give the judge an "Unknown" option for ambiguous cases, because forcing binary Pass/Fail on edge cases generates false positives you cannot distinguish from real ones.
One particularly valuable technique: run the evaluation 3 to 5 times and compute the variance between runs. A 3 percent metric shift across runs is noise, but 10 percent means something actually changed. Without this baseline, you are chasing random fluctuations and calling them regressions.
For agent evaluation specifically, the article introduces pass@k and pass^k metrics. pass@k tracks whether at least one out of k attempts succeeds, while pass^k tracks whether all k attempts succeed. As k grows, these tell opposite stories: pass@k climbs toward 100 percent while pass^k drops sharply, revealing how consistent your agent really is. This is a genuinely useful framing that deserves more attention than it typically gets.
The piece is part of a 7-article series, and it reads like solid coursework material. But there are some gaps worth noting. First, the entire framework assumes you have domain experts available to label data. For many teams, the bottleneck is not the evaluation methodology but the availability of qualified human labelers. Second, the article does not address the meta-problem: who evaluates the evaluator of the evaluator? At some point you are trusting human judgment, and human judgment has its own biases and inconsistencies. Third, the cost model for this iterative validation process is left completely unexamined. Running frontier models as judges, maintaining labeled datasets, and doing multiple iteration rounds adds up quickly.