You compare two agent versions with an LLM judge. The judge sees both transcripts and picks a winner. The new version wins 62% of the time, and you ship it. Then you find the judge picks whichever answer comes first more often than it should. The 62% is partly an artifact of the order you fed it.
This is position bias. It is well documented for pairwise LLM judges, and it is cheap to measure on your own judge, your own prompts and your own data. This tutorial gives you a swap test, a tie-aware scoring rule and a CI check. It is plain Python. It works with Langfuse, Promptfoo, Braintrust or pytest, because the logic sits around whatever calls the judge.
What you are measuring
Two separate things get confused:
- Position bias: the verdict depends on which candidate is labeled A.
- Verbosity bias: the verdict favors the longer answer regardless of correctness.
Position bias you can measure directly by swapping. Verbosity bias needs a length-controlled check, covered below. Neither is fixed by a better rubric alone. Measure first.
Step 1: run every pair in both orders
For each pair of agent outputs (x, y), call the judge twice: once as (A=x, B=y), once as (A=y, B=x). Store both raw verdicts.
# pairwise.py
from dataclasses import dataclass
@dataclass
class PairResult:
case_id: str
verdict_xy: str # "A", "B" or "TIE" with x in slot A
verdict_yx: str # same, with y in slot A
def judge_pair(judge, case, x, y) -> PairResult:
v1 = judge(case.task, a=x, b=y) # your judge call; returns "A" | "B" | "TIE"
v2 = judge(case.task, a=y, b=x)
return PairResult(case.id, v1, v2)
def resolve(r: PairResult) -> str:
"""Return 'x', 'y' or 'tie'. Disagreement across orders counts as a tie."""
first = {"A": "x", "B": "y", "TIE": "tie"}[r.verdict_xy]
second = {"A": "y", "B": "x", "TIE": "tie"}[r.verdict_yx]
return first if first == second else "tie"
The rule in resolve is the important part. If the judge says x in one order and y in the other, it did not judge the content. It judged the slot. Count that as a tie, not as a coin flip.
Step 2: compute the flip rate and the slot preference
def bias_report(results: list[PairResult]) -> dict:
n = len(results)
flips = sum(
1 for r in results
if r.verdict_xy in ("A", "B") and r.verdict_yx in ("A", "B")
and {"A": "x", "B": "y"}[r.verdict_xy] != {"A": "y", "B": "x"}[r.verdict_yx]
)
picks_a = sum(r.verdict_xy == "A" for r in results) + sum(r.verdict_yx == "A" for r in results)
decided = sum(r.verdict_xy != "TIE" for r in results) + sum(r.verdict_yx != "TIE" for r in results)
return {
"pairs": n,
"flip_rate": flips / n,
"slot_a_share": picks_a / decided if decided else None,
}
Read the two numbers like this:
flip_rateis the share of pairs where swapping the order changed the winner. For an unbiased judge on clear-cut pairs, it is near zero. Whatever you see is the fraction of your verdicts that carry no information.slot_a_shareis how often the judge picked slot A across all calls. Since each pair is shown in both orders, a content-driven judge lands near 50%. A value far from 50% is a slot preference.
We do not give a universal threshold, because the right one depends on how close your candidates are. Pick one from your own data: take 30 or more pairs where a human has labeled a clear winner, and see what flip rate the judge has on those. That is your floor.
Step 3: report win rate with ties shown
Report three numbers, not one: wins, losses, ties after resolution.
from collections import Counter
def summarize(results):
c = Counter(resolve(r) for r in results)
n = len(results)
return {k: c[k] / n for k in ("x", "y", "tie")}
If the new version "wins 62%" but ties are 40%, the headline is 62% of the 60% that were decided, plus a judge that could not separate the rest. That is a different launch conversation. Also run the noise check from is that eval improvement real before trusting a gap of a few points.
Step 4: check verbosity separately
Take your pairs and bucket them by length ratio, len(x) / len(y). Within each bucket, compare the judge's resolved wins to human labels. If the judge's accuracy against humans drops when the correct answer is the shorter one, you have verbosity bias. Two fixes that tend to help:
- Put "length is not a quality signal; extra detail that does not change the outcome is not a plus" in the judge prompt, and re-measure. Do not assume it worked.
- For agents, judge outcomes and trajectory steps rather than prose where you can. State correctness and tool-call correctness have no length to reward. See recording and replaying tool responses for making those checks deterministic.
Step 5: gate it in CI
Fail the run when the judge itself looks unreliable, before looking at who won.
# test_judge_bias.py
import json
MAX_FLIP_RATE = 0.15 # set this from your human-labeled floor, not from us
def test_judge_is_order_stable():
report = json.load(open("out/bias_report.json"))
assert report["pairs"] >= 50, "too few pairs to say anything"
assert report["flip_rate"] <= MAX_FLIP_RATE, report
assert 0.40 <= report["slot_a_share"] <= 0.60, report
The 0.15 and the 40-60 band are placeholders. Replace them with values derived from your labeled set and commit the derivation next to the test.
What this does not tell you
Swapping fixes order. It does not make the judge correct. A judge can be perfectly order-stable and consistently wrong. Agreement with human labels is still the number that matters, and the calibration procedure covers it. Judges also change under you when the model version changes, which is the case in your judge drifted and your pass rate lied. Rerun this bias report whenever the judge model or prompt changes.
Swapping doubles judge calls. For large suites, run the swap on a fixed sample each night and on every judge change, and use single-order verdicts on the rest only after the flip rate has stayed low.
If you want this wired into your harness with the thresholds set from your own labeled data, that is part of our eval suite engineering work.