Someone changes the planner prompt. The suite goes from 82% to 87%. The change ships.
Run the same suite again on the unchanged code and it might give you 85%. Agents are nondeterministic: sampling temperature, tool latency and ordering, retries, model-side routing, and an LLM-as-judge that is itself sampled. A pass rate is an estimate, and most teams treat it as a measurement. This tutorial covers how to find out how noisy your own suite is, how to compare two versions honestly, and how many runs you can afford in CI.
None of this requires a statistics background. It requires running the suite more than once.
Step 1: measure the flake rate before you measure anything else
Hold the code fixed. Run the whole suite N times. Any case that does not return the same verdict every time is flaky, and its flakiness is your noise floor.
# flake_check.py
from collections import defaultdict
N = 10
verdicts = defaultdict(list)
for _ in range(N):
for case in load_cases():
verdicts[case.id].append(run_case(case)) # True / False
for case_id, runs in sorted(verdicts.items()):
passes = sum(runs)
if 0 < passes < N:
print(f"{case_id}: flaky, {passes}/{N} passed")
Run this overnight before you argue with anyone about a number. Expect three groups:
| Group | What it means | What to do |
|---|---|---|
| Always passes | Stable | Keep |
| Always fails | A real, reproducible bug | Keep, fix the agent |
| Sometimes passes | Noise, or a genuine intermittent failure | Investigate before keeping |
The third group is the interesting one, and it splits again. Some cases are flaky because the agent is unreliable on that input — it picks the right tool 6 times in 10. That is a finding, not a defect in the suite; log the pass fraction as the score instead of collapsing it to a boolean. Others are flaky because the harness is sloppy: an unseeded fixture, a live tool that returns different data each run, a timestamp in the assertion, a judge with temperature above zero. Fix those. Every one you fix buys you statistical power for free, which is cheaper than more runs.
A suite where 15% of cases are harness-flaky cannot detect a 5-point regression. You will spend the rest of your time arguing about the wrong thing.
Step 2: put an interval on the pass rate
With flake under control, a single suite run gives you k passes out of n cases. Report that as an interval, not a point. For pass rates near 0 or 1 the normal approximation lies; use a Wilson interval.
from statsmodels.stats.proportion import proportion_confint
k, n = 82, 100
low, high = proportion_confint(k, n, alpha=0.05, method="wilson")
print(f"pass rate {k/n:.0%} (95% CI {low:.0%}-{high:.0%})")
# pass rate 82% (95% CI 73%-88%)
Eighty-two percent on a hundred cases means "somewhere between about 73% and 88%". That is the honest version. It also tells you what your dataset can and cannot resolve: with 100 cases, an improvement of three points is invisible. If you need to detect small movements, you need more cases, not more dashboards.
One caveat that matters more than the arithmetic: this interval describes sampling within your dataset. It says nothing about whether your dataset resembles production traffic. If the cases were invented in a planning meeting, a tight interval is a precise estimate of the wrong quantity. Sampling cases from real traces is a separate job and it comes first.
Step 3: compare versions with a paired test
The common mistake is comparing two pass rates as if they came from independent samples. They did not — both versions ran on the same cases. Pairing is what gives you power, so use it. For boolean verdicts, the right tool is McNemar's test, which looks only at the cases where the two versions disagreed.
from statsmodels.stats.contingency_tables import mcnemar
# per-case verdicts, same order, same cases
base = {c: run_case(c, version="base") for c in cases}
head = {c: run_case(c, version="head") for c in cases}
b = sum(1 for c in cases if base[c] and not head[c]) # newly failing
c_ = sum(1 for c in cases if not base[c] and head[c]) # newly passing
print(f"{c_} fixed, {b} broken")
print(mcnemar([[0, b], [c_, 0]], exact=True))
If 5 cases got fixed and 3 broke, the p-value will be nowhere near significant and you have learned the correct thing: the change did roughly nothing measurable, and it broke three cases you should go read. If 18 got fixed and 1 broke, you do not need a test to believe it, but the test costs nothing.
For flaky cases, run each case m times per version and compare pass fractions per case with a paired bootstrap instead of a single boolean. m=5 is usually enough to stop the noise dominating.
The number that should appear in the pull request is not the delta in pass rate. It is the two lists: cases that started failing, cases that started passing. A reviewer can read those. Nobody can review "+5%".
Step 4: decide what CI can afford
Running 200 cases five times each on every push is real money and real minutes. Split the suite:
- Per-PR gate: the deterministic trajectory and tool-call cases, one run each. These are cheap and near-zero flake, so a single failure is signal. This is the merge gate.
- Nightly: the full dataset, with judge-graded cases, 3-5 runs on anything known to be flaky, plus the paired comparison against yesterday's main.
- Pre-release: the full battery at higher m, plus a red-team re-run.
And set the alert threshold from the noise floor you measured in step 1, not from a round number someone liked. If a stable suite wanders by two points run to run, alerting at one point is a pager that everyone learns to ignore.
What this does not fix
A confidence interval will not tell you your judge is wrong. If the grader disagrees with your engineers 30% of the time, tightening the interval around its verdict just gives you a precise measurement of a bad instrument — go calibrate it first. Statistics also cannot rescue a dataset that misses the traffic that breaks you, and no p-value converts a pass rate into a launch decision; someone still has to weigh which failures are tolerable.
But the discipline is cheap. Measure the flake rate once. Report intervals. Compare paired, and show the fixed and broken lists. Do that and "the evals went up" stops being a claim anyone can make by re-running the suite until they like the number.
When we build a regression harness in eval suite engineering, this is part of the setup, not an afterthought: a suite you cannot trust to disagree with you is not evidence.