A team we worked with had 80 eval cases and a pass rate that climbed from 61% to 94% over six weeks of prompt edits. Production complaint volume did not move. Nothing was faked. Every one of those 80 cases was fixed by a real change to the prompt or the tool schemas. The problem is that the engineer editing the prompt could see all 80 failures, and a human in a loop with a scoreboard is a gradient descent step. After enough iterations, the suite stopped measuring whether the agent handles the task and started measuring how well the prompt had been fitted to 80 specific transcripts.
This is the oldest problem in machine learning wearing a new hat, and agent teams keep walking into it because eval cases feel like unit tests. Unit tests do not degrade when you optimize against them; a test that asserts 2 + 2 == 4 is correct forever. An eval case is a sample of a distribution you care about, and its value depends on the agent not having been tuned to that sample.
The fix is a holdout split. It costs about an hour to set up and it is the difference between a pass rate you can quote to your head of engineering and one you cannot.
Step 1: split the dataset, and split it by scenario
Randomly assigning cases to dev and holdout is the obvious move and it is usually wrong for agents. Agent datasets are full of near-duplicates: the same refund scenario with three different order IDs, twelve variants of one injection payload. Split those randomly and the holdout contains cousins of dev cases, which leaks the answer and makes the gap look smaller than it is.
Split by scenario group instead. Tag each case with the failure mode or user journey it came from, then assign whole groups to one side.
import hashlib
def split_for(scenario_group: str, holdout_frac: float = 0.3) -> str:
"""Stable, reproducible group-wise split. Same group always lands the same side."""
h = int(hashlib.sha256(scenario_group.encode()).hexdigest()[:8], 16)
return "holdout" if (h % 100) / 100 < holdout_frac else "dev"
Hashing rather than shuffling matters: the assignment is reproducible from the group name alone, so two engineers on two machines get the same split without committing a mapping file. Aim for roughly 70/30. Below about 40 holdout cases the pass rate is too noisy to read a gap off it at all, which is the arithmetic in measuring noise in agent pass rates.
If you mined the dataset from traces, as in building an agent eval dataset from production traces, you already have the grouping key: the failure taxonomy label you assigned during trace review.
Step 2: carry the split as dataset metadata
Put the split on the case, not in a filename. Filenames get reorganized; metadata travels with the row and survives being copied between tools.
In Langfuse, dataset items take arbitrary metadata:
from langfuse import Langfuse
langfuse = Langfuse()
langfuse.create_dataset_item(
dataset_name="support-agent-v3",
input={"query": "Refund order 4832, it arrived cracked"},
expected_output={"tool_sequence": ["lookup_order", "issue_refund"]},
metadata={
"scenario_group": "refund-damaged-item",
"split": "holdout",
"added": "2026-09-02",
},
)
Then filter at run time when you iterate the dataset, and record the split on the run name so the two numbers never get averaged together by accident:
dataset = langfuse.get_dataset("support-agent-v3")
for item in dataset.items:
if item.metadata.get("split") != "dev":
continue
with item.run(run_name="prompt-v18-dev") as root_span:
result = run_agent(item.input["query"])
root_span.score_trace(name="tool_sequence_match", value=score(result, item.expected_output))
In Promptfoo the same idea is a tag plus a filter. Add metadata: {split: holdout} to each test and select at the command line:
npx promptfoo eval --filter-metadata split=dev
npx promptfoo eval --filter-metadata split=holdout --output holdout.json
In plain pytest, a marker does it:
pytest evals/ -m "not holdout" # daily loop
pytest evals/ -m holdout # release gate only
Step 3: give the two sets different jobs
The split only works if the access rules differ. Write them down, because this is a discipline problem more than a tooling problem.
| Dev set | Holdout set | |
|---|---|---|
| Who reads individual failures | anyone, constantly | one reviewer, at release time |
| Run frequency | every commit, in CI | release candidates and weekly |
| Allowed to drive prompt edits | yes | no, only to accept or reject |
| Reported to stakeholders | no | yes |
| Rotation | grows freely | replaced on a schedule |
The hard rule is the third row. When the holdout drops, you are allowed to know that it dropped and which scenario group moved. You are not allowed to open the failing transcripts, hand them to the prompt, and re-run until green. Do that twice and the holdout is a dev set with extra ceremony.
The legitimate move when holdout regresses is to reproduce the failure mode as new dev cases written from first principles — same failure class, fresh inputs — fix those, and then re-check the holdout. It is slower. It is also the only version that keeps the number honest.
Running the holdout only at release time has a side benefit: it is the expensive suite. If the dev set runs on every commit and the holdout runs weekly, you can afford more turns, more seeds and more model-graded rubrics in the holdout without anyone complaining about CI minutes.
Step 4: read the gap
Report both numbers together, always, with the case counts:
prompt-v18 dev 94% (n=112) holdout 71% (n=48) gap 23pt
A gap is not automatically a scandal. Some gap is sampling noise; with 48 holdout cases, a couple of points either way is nothing. What you are watching is the trend in the gap across releases.
- Gap stable and small (under ~5pt). The suite is measuring the task. Keep going.
- Gap growing while dev climbs. Classic overfitting. The prompt is accumulating clauses that only help specific dev cases. Look for scenario-specific instructions in the system prompt — literal order IDs, a rule about cracked items — and delete them.
- Gap large from day one. Not overfitting; the splits are not drawn from the same distribution. Usually the dev set was hand-written and the holdout was mined from traces, or vice versa. Rebuild both from the same source.
- Holdout above dev. Check for a leak: a dev case whose expected output is wrong, or a group that landed on both sides because two people labeled the same scenario differently.
One caveat that bites: your grader can overfit too. If you have been tuning an LLM-as-judge rubric against dev failures, the judge may have learned dev's quirks. Hold out a slice of human labels for the judge as well, and re-measure agreement on that slice when you change the rubric. That is the same measurement described in an unmeasured LLM-as-judge is just a guess, applied to the judge's own dataset.
Step 5: rotate, without losing regression coverage
A holdout set is consumed slowly no matter how careful you are. Engineers see aggregate failures, remember which scenarios hurt, and write prompts with those in mind. Assume a holdout has a shelf life of a quarter or so.
Rotation is not deletion. When a holdout case has been leaked into the dev loop, move it to a third bucket — call it regression — and keep running it forever as a pass/fail guard. It no longer contributes to the headline number, because it is no longer an unbiased sample, but it still catches the day someone breaks refunds entirely.
split=dev iterate freely, drives development
split=holdout unbiased estimate, reported externally
split=regression retired holdout, must never fail, excluded from the estimate
Then refill the holdout from new production traces on the same cadence you mine traces anyway. Fresh traffic is the only honest source of new holdout cases: it contains the scenarios nobody on the team has thought about, which is exactly the property you are paying for.
What this does not tell you
A healthy dev/holdout gap says your suite generalizes across the cases you sampled. It says nothing about scenarios absent from both sets, and nothing about adversarial behavior — an attacker is not sampling from your traffic distribution, which is why red-teaming an agent is not a pentest and why red-team cases live in their own suite rather than in this split.
If you want the short version: the first thing we do on an Agent Readiness Assessment when a team shows us a 90-plus pass rate is ask which cases the prompt was written against. If the answer is "all of them," the number tells us how long the team has been iterating, not whether the agent works. Splitting the set costs an hour and makes the number mean something.