Here is a bug we get called about a few times a year. The agent pass rate drops four points overnight. Nobody merged a prompt change, the model pin did not move, the dataset is the same. The team spends two days bisecting the agent. The agent was fine. The grader changed.
An LLM-as-judge is a model in your test harness. It has a version, a prompt, a temperature and a provider who ships updates on their schedule, not yours. Calibrating it once tells you it agreed with your humans in March. It says nothing about today. This tutorial builds the missing piece: a regression suite for the judge itself, so the grader is monitored with the same suspicion you point at the agent.
Four things that move a judge
Drift is not one failure. Separating the causes decides what you do about it.
| Cause | How it shows up | What it means |
|---|---|---|
| Provider rotated the model behind an alias | agreement drops, no commit touched the harness | pin the version, re-measure, re-baseline |
| Someone edited the rubric | drop starts exactly at one commit | expected; re-baseline deliberately |
| Dataset shifted under a stable judge | agreement holds, pass rate moves | the agent or the traffic changed, not the judge |
| Label rot | humans now disagree with their own old labels | your rubric was ambiguous; fix the rubric first |
The first two are judge problems. The third is the signal you actually wanted. The fourth is the most common and the least fun: if two reviewers cannot reproduce a label six weeks later, the judge was never the weak link.
Step 1: stop using floating model aliases in the harness
A grader pointed at a rolling alias is a grader that can change without a commit. Pin the exact snapshot, and record the pin in the result row so an old score can be explained a year from now.
# judge.py
JUDGE = {
"model": "gpt-4.1-2025-04-14", # snapshot id, never a rolling alias
"temperature": 0,
"rubric_version": "refund-tone-v3",
}
def grade(case, agent_output):
prompt = render_rubric(JUDGE["rubric_version"], case, agent_output)
verdict = call_model(JUDGE["model"], prompt, temperature=JUDGE["temperature"])
return {
"pass": verdict["pass"],
"reason": verdict["reason"],
"judge_model": JUDGE["model"],
"rubric_version": JUDGE["rubric_version"],
}
Store the rubric text in the repo, versioned, not in a vendor UI. A rubric edited in a web form is an untracked change to your test suite. That applies whether the runner is Promptfoo, Braintrust or a hundred lines of pytest.
Pinning does not stop drift. Providers deprecate snapshots, and you will eventually be forced onto a new one. What pinning buys you is that the move happens on a date you chose, with a measurement attached.
Step 2: freeze a gold set and never grow it casually
The gold set is the judge's test data: agent outputs with human labels attached, held fixed so that any change in judge verdicts is attributable to the judge.
What has worked for us:
- 80 to 150 cases. Small enough that two humans can re-label it in an afternoon, big enough that a two-case flip is not 10% of the signal.
- Stratified on purpose. Roughly a third clear passes, a third clear failures, a third the arguments you had during calibration. A gold set of easy cases cannot detect drift, because every judge gets easy cases right.
- Two independent human labels per case, with disagreements resolved by writing down a rule and putting that rule in the rubric.
- Frozen. New cases go to a staging file and get merged at a named version bump, with baselines recomputed once, on purpose.
{"id": "g-014", "input": "...", "agent_output": "...", "human_label": "fail", "note": "promises a delivery date"}
{"id": "g-015", "input": "...", "agent_output": "...", "human_label": "pass", "note": "hedged correctly"}
Commit it next to the agent code. Like every other deliverable we ship, it belongs in your repository, not in ours.
Step 3: measure agreement, not the judge's own pass rate
The judge's pass rate on the gold set can stay flat while the judge quietly swaps which cases it passes. Measure agreement against the human labels, and correct for chance. Cohen's kappa is enough here:
def kappa(human, judge):
n = len(human)
agree = sum(h == j for h, j in zip(human, judge)) / n
# expected agreement if both labelled at their own base rates
ph = sum(h == "pass" for h in human) / n
pj = sum(j == "pass" for j in judge) / n
chance = ph * pj + (1 - ph) * (1 - pj)
return (agree - chance) / (1 - chance)
Record four numbers per run, not one: kappa, raw agreement, false-pass rate and false-fail rate. The last two matter asymmetrically. A judge that passes bad agent output is a release-blocker you disabled without noticing; a judge that fails good output costs engineering time and credibility. Track them separately or you will average away the dangerous one.
While you are there, run the judge twice on the same gold set with temperature 0 and compare. Non-zero self-disagreement is your noise floor, and it caps how small a drift you can detect at all. The same logic as measuring noise in agent pass rates applies one layer up.
Step 4: make it a scheduled test with a baseline file
Commit the baseline, then fail loudly when the live measurement leaves it.
# test_judge_drift.py
import json
BASELINE = json.load(open("judge_baseline.json"))
# {"kappa": 0.79, "false_pass": 0.04, "flips": {}}
def test_judge_agreement_holds():
gold = load_gold_set()
verdicts = [grade(c, c["agent_output"])["pass"] for c in gold]
human = [c["human_label"] == "pass" for c in gold]
k = kappa(["pass" if h else "fail" for h in human],
["pass" if v else "fail" for v in verdicts])
false_pass = sum(v and not h for v, h in zip(verdicts, human)) / len(gold)
assert k >= BASELINE["kappa"] - 0.05, f"judge agreement fell to {k:.2f}"
assert false_pass <= BASELINE["false_pass"] + 0.02, "judge is passing bad output"
def test_no_unexplained_flips():
"""Which cases flipped is more diagnostic than how many."""
flips = [c["id"] for c, v in zip(load_gold_set(), current_verdicts())
if BASELINE["verdicts"].get(c["id"]) not in (None, v)]
assert not flips, f"judge changed its mind on {flips}"
Run it nightly and in the pull request that touches the rubric, the judge config or the harness. Nightly catches the provider; the pull request catches you. Both write their numbers into the same history table, keyed on judge model and rubric version, so a step change lines up with a cause. If your scores already live in Langfuse, this is one more named score on a fixed dataset — the mechanics are in scoring agent traces in Langfuse.
One rule about the assertion thresholds: set them from your measured noise floor, not from a round number you like. A 0.05 kappa band is a starting guess. If two temperature-0 runs already disagree by 0.04, that band is theatre.
Step 5: have a written response, because the alert will fire
When the drift test goes red, the wrong move is to widen the threshold. What we do instead, in order:
- Re-run the judge twice. Rule out noise before rule out drift.
- Diff the flipped cases. Five cases all in one rubric clause is a rubric problem. Five unrelated cases is a model problem.
- Check what changed outside the repo. Snapshot deprecation notices, provider status pages, a rubric edited in a vendor UI, an SDK bump that changed a default.
- Re-label a sample of the flips by hand. Sometimes the new judge is right and the old labels were wrong. That is a finding, not an emergency.
- Quarantine downstream numbers. Any agent pass rate graded between the last green drift test and now is suspect. Say so before someone quotes it in a launch review.
- Re-baseline deliberately, in a commit, with the new judge pin and a note on why the number moved.
Step 5 is the one teams skip and the one that costs the most. A launch decision made on a silently mis-graded pass rate is worse than no pass rate, because it came with confidence attached.
What this does and does not buy you
It buys you attribution: when a number moves you can tell whether the agent, the traffic or the grader moved. It does not make the judge correct, and it does not extend to rubric clauses your gold set never exercises. If you add a clause, add cases for it and re-baseline, or you are monitoring last quarter's rubric.
Judge drift checks are part of the ongoing reliability engineering we run alongside eval suites, next to dataset growth and red-team re-runs. If you are standing this up yourself, the order that matters is: pin the model, freeze the gold set, measure agreement and the two error directions separately, then alert on flips. If you would rather have someone else build and run it in your repo, tell us about the agent and an engineer will reply within one business day.