Somewhere in most agent eval suites there is a grader that asks a model "Was this response helpful and accurate? Answer 1-5." It returns 4.2 on average. The dashboard is green. Nobody has checked whether a human would agree.
This is the most common weakness we find when assessing a suite, and it is fixable in about a week.
Why model-graded evals are unavoidable
Agents produce outputs that cannot all be checked exactly: a drafted email, a summary of retrieved documents, an explanation to a user. For these, an LLM-as-judge is the practical tool. The mistake is not using one; it is treating its scores as measurements without ever measuring the judge.
Everything that can be checked deterministically should be. Tool selection, argument values, call ordering, state after the run, schema validity, length and cost budgets: these are exact comparisons and never drift. Reserve the judge for the fuzzy remainder, and then calibrate it.
Step 1: build a labeled set
Sample at least 100 cases from the population the judge will grade, stratified so the hard cases are represented rather than the easy majority. Have two humans who know the product label each one independently against the same rubric the judge will use. Record their agreement with each other first. If two people cannot agree on what a 4 means, the judge has no chance and the rubric is the problem.
Aim for a rubric where the human-human agreement is high before you touch the model.
Step 2: choose the metric
Percentage agreement is easy to read and easy to fool: if 90% of cases are passes, a judge that always says "pass" scores 90%. Use a chance-corrected statistic (Cohen's kappa for two raters, or Krippendorff's alpha for more) alongside raw agreement, and look at the confusion matrix. The interesting failures are always in the off-diagonal cells.
For binary graders, report precision and recall separately. A judge that catches every bad response but flags a third of the good ones will get the suite ignored within a month.
Step 3: measure and read the disagreements
Run the judge over the labeled set. Do not stop at the number. Read every case where judge and humans disagree and sort them into buckets:
- Rubric ambiguity. Humans read the rubric differently from the model. Rewrite the rubric with an explicit definition and two examples per score level.
- Missing context. The judge did not see the retrieved documents or the tool results, so it could not check faithfulness. Give it the same evidence the human had.
- Position and length bias. Judges favor longer responses and, in pairwise setups, the first option. Randomize order and add an explicit instruction that length is not quality.
- Self-preference. A judge grading outputs from its own model family rates them higher. Use a different model for the judge, or at minimum check for this.
- Genuinely hard. The humans disagreed too. Either sharpen the rubric or accept that this slice is noise and exclude it from thresholds.
Iterate the rubric, re-run, re-measure. Two or three rounds usually gets a well-posed judge above 85% raw agreement with a healthy kappa.
Step 4: decide whether to keep the judge
If after three rounds the judge still disagrees with humans on a material share of cases, the honest move is to stop. Either the question is not answerable from the evidence available, or it can be decomposed into smaller checks that are. "Was this a good response?" is usually five questions in a trench coat: did it answer, was it grounded in the retrieved content, did it follow the format, did it avoid the forbidden actions, was it the right length. Four of those are deterministic.
Step 5: guard against drift
A judge calibrated in March is not calibrated in September. The judge model gets updated, the agent's outputs shift as its prompt changes, and the rubric that once covered the failure modes no longer does.
- Pin the judge model version and record it with every run.
- Re-label a fresh sample of 50 cases every month and re-measure agreement.
- Track the judge's score distribution over time; a sudden shift with no code change is the judge, not the agent.
- Keep a small set of "anchor" cases with known scores and fail the run if the judge drifts on them.
What to publish
Every judge in the suite should have, next to its definition, the date it was last calibrated, the sample size, the raw agreement and kappa, and the model version. When leadership asks how much to trust a green dashboard, that table is the answer. Without it, the number is a guess wearing a decimal point.