A vendor sends you a chart. Their agent scores 68% on tau-bench retail, 54% on GAIA, some number on SWE-bench Verified. Your head of product asks whether that means the support agent you are about to launch is ready. It does not. Public benchmarks measure a different task, on a different tool surface, with a different definition of success, usually at k=1.
That does not make them useless. The benchmarks worth reading are worth reading for their method, not their leaderboard. This tutorial takes the two ideas from agentic benchmarks that transfer to your own agent - database-state scoring and pass^k reliability - and shows how to run them on your task, in a few hundred lines.
Why the leaderboard number does not transfer
Four reasons, in rough order of how much damage they do.
Different task distribution. tau-bench retail is order lookups, exchanges and returns against a fixed mock API with about a dozen tools. Your agent has your tools, your schema, your half-documented internal service that returns 200 with an error in the body. Nothing about a 68% on their traffic predicts a number on yours.
Different success criterion. tau-bench compares final database state and checks required outputs; SWE-bench runs the repo's test suite; GAIA checks an exact answer string. Your definition of "done" is probably softer and more argued-about than any of those. If you have not written yours down, the benchmark comparison is the least of your problems.
Contamination and age. Any benchmark published long enough to appear in a sales deck has had its tasks, solutions and discussion scraped into pretraining. Scores drift up over time without the underlying capability moving as much. Treat an old public number as an upper bound at best.
k=1 reporting. Most published headline numbers are a single attempt per task. Agents are stochastic. The number you actually care about in production is not "can it do this once" but "does it do this every time".
That last one is the fixable one, and it is where tau-bench's pass^k metric earns its keep.
pass^k: the metric that matches what you promise
pass@k, from the code-generation literature, asks whether at least one of k samples succeeds. That is the right metric when a human picks the good one. It is the wrong metric for an agent acting on its own.
pass^k inverts it: run each task k times and score the fraction of tasks that succeed on all k attempts. One flaky run fails the task. It is the metric that matches the promise you are making when you let an agent issue a refund without a human in the loop.
The estimator, given c successes out of n trials for a task, is the probability that a random k-subset is all successes:
# passhat.py
from math import comb
from statistics import mean
def pass_hat_k_task(n_trials: int, n_success: int, k: int) -> float:
# Unbiased estimate that k independent runs of this task all succeed.
if k > n_trials:
raise ValueError("need at least k trials per task")
if n_success < k:
return 0.0
return comb(n_success, k) / comb(n_trials, k)
def pass_hat_k(results: dict[str, list[bool]], k: int) -> float:
# results: task_id -> list of per-trial pass/fail
return mean(
pass_hat_k_task(len(trials), sum(trials), k) for trials in results.values()
)
Run n=8 trials per task, then report the curve, not one number:
results = run_suite(tasks, trials=8) # your harness; see below
for k in (1, 2, 4, 8):
print(f"pass^{k} = {pass_hat_k(results, k):.2f}")
A healthy suite decays gently: pass^1 = 0.88, pass^8 = 0.81. A suite that reads pass^1 = 0.88, pass^8 = 0.41 is telling you that half your "passing" tasks are coin flips, and that the single-attempt number on your dashboard is fiction.
Two caveats. The trials have to be independent: fresh database state, fresh conversation, fresh memory, or you are measuring caching. And eight trials per task multiplies your eval bill by eight, so run the full k-curve nightly or pre-release and keep CI on k=1 plus a small flaky-watchlist subset. The related problem of telling a real improvement apart from sampling noise is covered in measuring noise in agent pass rates.
Borrow the scoring method, not the tasks
The other transferable idea is how tau-bench decides a task passed: it does not grade the transcript. It diffs the world.
Give each task an initial state, a user goal, and an expected final state, and score by comparing the database after the run against the expected database, ignoring fields the task does not care about.
# harness.py
from copy import deepcopy
from dataclasses import dataclass, field
from typing import Any
@dataclass
class Task:
id: str
initial_db: dict[str, Any]
user_goal: str # what the simulated user wants
expected_db: dict[str, Any] # only the keys this task asserts on
required_outputs: list[str] = field(default_factory=list)
forbidden_tools: list[str] = field(default_factory=list)
def db_subset_matches(actual: dict, expected: dict) -> bool:
# Compare only the keys the task cares about, recursively.
for key, want in expected.items():
got = actual.get(key)
if isinstance(want, dict):
if not isinstance(got, dict) or not db_subset_matches(got, want):
return False
elif got != want:
return False
return True
def score(task: Task, run) -> bool:
if not db_subset_matches(run.final_db, task.expected_db):
return False
text = run.transcript_text.lower()
if any(s.lower() not in text for s in task.required_outputs):
return False
called = {c.name for c in run.tool_calls}
return not (called & set(task.forbidden_tools))
Three properties make this worth copying:
- No judge in the loop for the part that matters. State equality is deterministic, so there is no grader to calibrate and no judge drift. Save the LLM-as-judge for tone and explanation quality, where it belongs.
- Partial-key comparison. Asserting the whole database makes every task brittle. Assert the row the task is about.
- Negative assertions are cheap.
forbidden_toolscatches the refund that was issued along the way to the right answer.
A sketch of the trial loop, with the per-trial reset that keeps pass^k honest:
def run_suite(tasks: list[Task], trials: int = 8) -> dict[str, list[bool]]:
results: dict[str, list[bool]] = {}
for task in tasks:
outcomes = []
for _ in range(trials):
env = Environment(db=deepcopy(task.initial_db)) # fresh state each trial
run = agent.run(user=SimulatedUser(task.user_goal), env=env)
outcomes.append(score(task, run))
results[task.id] = outcomes
return results
The simulated-user half of this is its own piece of work; the mechanics are in multi-turn agent evals with a simulated user, and keeping external tools deterministic across eight trials is tool replay.
What to do with a vendor's benchmark claim
You will still be handed leaderboard numbers. Four questions make them tractable:
- At what k, and over how many trials? If the answer is "one run", the number is an optimistic ceiling.
- Scored how? State comparison, exact match, or a model grading free text? If a judge scored it, ask for the human-agreement number. There usually is not one.
- When was the benchmark published, and when were the task sets last rotated? Old set, high score, assume contamination.
- Can it be re-run on our tasks? This is the only question whose answer is worth anything. Twenty of your own tasks, scored by state diff, beat any public suite for deciding whether to launch.
If the vendor cannot run on your tasks, run it yourself. Wrapping a candidate model or framework in the harness above and getting a real number on twenty of your own cases is about a day of work once the fixtures exist.
What this does not tell you
A domain benchmark with a healthy pass^k curve says your agent completes your happy-path and known-edge-case work repeatably. It says nothing about what happens under adversarial input: a poisoned tool result or a user probing an authorization boundary is a separate battery, and it is not a pentest either. It also says nothing about tasks you did not think to write down, which is why the dataset has to keep growing from production traces.
Still: twenty tasks, eight trials, state-diff scoring and a pass^k curve is a defensible answer to "is it ready". A leaderboard screenshot is not. Building that harness on a client's own stack is most of what our eval suite engineering work is, and a first version of it ships inside a readiness assessment.