Every eval suite we are asked to fix has the same origin story. Someone wrote twelve test cases by hand in an afternoon, the suite passed, and then the agent failed in production on an input nobody imagined. The suite was not wrong. It was unrepresentative.
You already have a better source of cases than your imagination: the traces your agent produced last month. This tutorial turns them into a versioned eval dataset — sampled on purpose, deduplicated, labeled, split into a working set and a holdout, and committed to the repo as a file a reviewer can read.
We are using Langfuse here because it is common and the export is straightforward, and the scoring tutorial covers the SDK basics. The method is the same for Braintrust, LangSmith, or a table of JSON blobs in Postgres.
Step 0: prerequisites
A month or so of agent traces with the tool calls recorded, pip install langfuse, and somewhere in the repo to put the result — we use evals/data/. You also need to know your traffic's shape well enough to name three or four segments of it: request type, customer tier, channel, whatever your product actually varies on.
Step 1: pull the raw traces
from langfuse import Langfuse
langfuse = Langfuse()
page, traces = 1, []
while True:
batch = langfuse.api.trace.list(
from_timestamp="2026-07-01T00:00:00Z",
to_timestamp="2026-08-01T00:00:00Z",
page=page,
limit=100,
)
if not batch.data:
break
traces.extend(batch.data)
page += 1
print(len(traces), "traces")
Do this once, write the raw result to disk, and never call the API again during development. Everything downstream is a pure function of that file, which is what makes the dataset reproducible when someone asks in November where case 34 came from.
Strip the payloads down to what a case needs — input, tool calls, final output, and a few segment fields — and drop everything else. A case file full of internal ids and timestamps is a case file nobody reads.
Step 2: give every trace a signature so you can dedupe
Production traffic is repetitive. If 40% of your traces are "where is my order", uniform random sampling gives you a dataset that is 40% one intent, which wastes the cases you can afford to grade.
Dedupe on behavior, not on wording. The cheapest signature that works is the tool-call shape:
def signature(trace):
calls = [c["name"] for c in trace["tool_calls"]]
# collapse consecutive repeats: lookup,lookup,refund -> lookup,refund
collapsed = [n for i, n in enumerate(calls) if i == 0 or n != calls[i - 1]]
return "|".join(collapsed) or "no_tools"
Group by signature and you get a frequency table of what your agent actually does. That table is useful on its own: the long tail of signatures with a count of one or two is where the interesting failures live, and the head is where your regression risk lives. Keep both.
If you need finer granularity than tool shape, add a normalized intent — an embedding cluster or a cheap classifier over the first user message — but do that only after the tool-shape table stops separating cases.
Step 3: sample on purpose
Now stratify. Take a fixed quota from each bucket rather than sampling the whole pool at random:
import random
from collections import defaultdict
random.seed(1234) # the seed is part of the dataset
buckets = defaultdict(list)
for t in traces:
buckets[(t["segment"], signature(t))].append(t)
cases = []
for key, group in buckets.items():
quota = 3 if len(group) >= 3 else len(group)
cases.append(random.sample(group, quota))
Set the seed, commit the seed. "Rerun the sampler" should produce the same dataset for anyone on the team.
Three sources deserve a quota of their own, above what frequency alone would give them:
| Source | Why it earns a quota |
|---|---|
| Escalations and thumbs-down | Known-bad behavior with a label attached |
| Rare tool signatures | Where the agent improvises, and where it breaks |
| Anything that touched an irreversible tool | Refunds, sends, deletes — the failures that cost money |
Fifty to eighty cases is a reasonable first target. That is the same 50+ starter set we build during a readiness assessment, and it is enough to detect a real regression while staying small enough that a human can read every case once.
Step 4: know what a pass rate on 60 cases can and cannot tell you
This is where teams over-read their own numbers. A pass rate from a finite sample is an estimate with a width. At 60 cases and 55 passes, the Wilson 95% interval on the true pass rate runs roughly from 0.83 to 0.96. So a move from 90% to 93% next week is not news.
from statsmodels.stats.proportion import proportion_confint
lo, hi = proportion_confint(55, 60, alpha=0.05, method="wilson")
print(round(lo, 3), round(hi, 3))
Two practical consequences. First, report intervals next to pass rates in any document that leads to a launch decision. Second, gate CI on named cases that must not fail — the irreversible-action ones especially — rather than on a headline percentage, because at this sample size the percentage moves on noise. The CI gating tutorial goes through that comparison logic.
If you need to resolve small differences, you need more cases, and the cost of more cases is grading. That is the actual budget constraint, not compute.
Step 5: label the expectation, not the old output
The trace tells you what the agent did. It does not tell you what it should have done. Somebody has to decide that, and it has to be somebody who knows the domain.
Do not paste the historical output in as the expected answer. You will freeze current behavior, bugs included, and every future improvement will read as a regression.
Write the expectation as assertions instead:
- id: case-034
source_trace_id: 8f2c...
segment: refunds
input: "I got two of the same lamp, send one back"
expect:
must_call: [lookup_order]
must_not_call: [issue_refund]
max_tool_calls: 6
rubric: "Explains the return process and does not promise a refund amount."
notes: "Duplicate shipment, not damage. Agent refunded on 2026-07-14."
Most rows should be deterministic checks on the trajectory. Use a rubric line only for what is genuinely fuzzy, and remember that a rubric grader is worthless until you have measured its agreement with a human — see calibrating an LLM judge.
Two people should label the first twenty cases independently. Where they disagree, the spec is ambiguous, not the labeler. Fix the spec.
Step 6: split, freeze, and let it grow
Hold back 20-30% of cases as a set you do not look at while iterating on prompts. Without a holdout you will tune the agent to the cases and learn nothing about the next input.
Then treat the dataset like code:
- One file per split, in the repo, reviewed by pull request.
- Every case carries its source trace id and its labeling date.
- Changing an expectation is a commit with a reason in the message.
- Re-run the sampler monthly on new traffic and add cases from any incident, so the dataset tracks reality instead of aging into it.
Dataset growth from production traces, judge drift checks and regression triage are the recurring parts of ongoing reliability work. None of it is complicated. It is just work that nobody schedules until an eval suite passes on the day the agent visibly fails.
Start with sixty honest cases, sampled and labeled the way above. It will tell you more about whether your agent is ready than a thousand you made up.