Judgeworthy

Does your judge just like long answers? A length-bias test for LLM graders

A common way an agent eval lies: the agent gets more verbose after a prompt change, the judge scores it higher, and the pass rate goes up. Correctness did not change. The grader rewarded length.

We have written about position bias in pairwise judges and judge drift. This is the third common grader defect: verbosity bias in single-answer scoring. It takes about an afternoon to measure.

What we will and will not claim

The test tells you whether your judge, with your rubric, on your task, moves its score when you add content-free text. It does not tell you the judge is correct. Run it alongside a human-agreement check, not instead of one.

Step 1: build a padded twin for each case

Take 40 to 60 graded cases where you already have a human label. For each answer, create a padded twin that adds length but no information. Two cheap transforms work:

  • Restatement. Prepend a paragraph that restates the user's question.
  • Hedged summary. Append a paragraph that summarizes the answer in different words.

Generate the padding with a model, then check by hand on a sample of ten that no new claims or tool results slipped in. If the padding adds facts, the test is invalid.

PAD_PROMPT = """Rewrite the answer below with an added opening paragraph that restates the question and a closing paragraph that summarizes the answer. Add NO new facts, numbers, or actions. Do not change the original text.

Question: {question}
Answer: {answer}"""

def make_twin(llm, case):
    return llm(PAD_PROMPT.format(question=case["question"], answer=case["answer"]))

Step 2: score both versions, same judge, same settings

Use the exact grader config you run in CI: same model, same temperature, same rubric. Score each original and each twin. Run each three times and average, so sampling noise does not look like bias. (See measuring noise in pass rates.)

import statistics as st

def mean_score(judge, case, answer, n=3):
    return st.mean(judge(case["question"], answer, case["rubric"]) for _ in range(n))

deltas = []
for c in cases:
    base = mean_score(judge, c, c["answer"])
    padded = mean_score(judge, c, c["twin"])
    deltas.append(padded - base)

print("mean delta", st.mean(deltas))
print("share of cases scored higher when padded", sum(d > 0 for d in deltas) / len(deltas))

Step 3: read the result

Two numbers matter.

  • Mean delta. If the padded twin does not change meaning, the mean should sit near zero. A consistent positive delta is the bias.
  • Flip rate. Count cases where the pass/fail decision changes between original and twin. A flip on a content-free edit is a defect. Report it as flips out of N, with the N.

With 50 cases, a small delta is within noise. Bootstrap a confidence interval over the per-case deltas before you act on it.

import random

def boot_ci(xs, reps=2000):
    means = sorted(st.mean(random.choices(xs, k=len(xs))) for _ in range(reps))
    return means[int(0.025 * reps)], means[int(0.975 * reps)]

print(boot_ci(deltas))

If the interval excludes zero, treat the bias as real.

Step 4: check the reverse direction

Run the opposite twin too: a concise rewrite of correct answers that keeps every fact. If the judge penalizes the short version, your rubric is rewarding thoroughness-by-volume. Concise correct answers failing is the more expensive error in production, because it pushes your agent toward padding.

Step 5: fix the rubric, then re-measure

Fixes that tend to work, in order of cost:

  1. Score against required facts, not overall quality. List the facts or tool results the answer must contain, and have the judge check each as yes or no. Open-ended quality scales invite length bias.
  2. State it in the rubric. Add: "Length is not evidence of quality. Do not reward restatement or summary." Then re-run. Do not assume it worked.
  3. Grade the trajectory, not the prose. For agents, the tool calls and final state are often checkable without a judge at all.
  4. Add a length penalty or cap only as a last resort. It hides the defect rather than removing it.

Re-run steps 2 and 3 after each change and keep the before and after numbers in the repo.

Step 6: keep it as a regression test

Commit the padded twins as a fixed fixture. Run the delta check whenever you change the judge model, the judge prompt or the rubric. Fail the build if the interval on the mean delta excludes zero or the flip rate exceeds a threshold you set from your first measurement.

Limits

  • Padding with a model leaves stylistic fingerprints. A judge may react to the style, not the length. Hand-written padding on a small subset is a useful cross-check.
  • This tests one bias. Judges can also favor their own model family's phrasing. Test that separately.
  • Passing this test does not make a judge accurate. Measure agreement with human labels as well.

If you want a second pair of eyes on a grader, our Eval Suite Engineering work includes calibrating judges against human labels and committing the checks to your repo.