Most advice on eval datasets starts the same way: mine your production traces. Good advice, and we have written the procedure. It is useless the week before launch, when the agent has handled forty runs, thirty-eight of which were you typing into a dev console.
This is the cold-start problem. You need a dataset to decide whether to ship, and shipping is how you get the data for the dataset. The way out is not to wait. It is to build a deliberately constructed starter set — ours are 50+ cases — from the two things you already have: the agent's tool schemas and a failure taxonomy. Then you replace it with real traffic as soon as real traffic exists.
This tutorial is the generation procedure. It is mostly ordinary code plus one LLM step, and the LLM step is the part you trust least.
What a starter case is
A case is not a prompt. It is a prompt plus the conditions the run happens under plus what must be true afterwards. Write them as data from the start:
# cases/refund_happy_path.yaml
id: refund-001
input: "I got order 4832 and the mug is cracked. Can I get my money back?"
fixtures:
lookup_order:
4832: { status: "delivered", total: 24.00, refundable: true }
expect:
must_call: ["lookup_order", "issue_refund"]
must_not_call: ["cancel_order"]
ordered: [["lookup_order", "issue_refund"]]
max_tool_calls: 6
rubric: "Confirms the refund amount and does not promise a delivery date."
The fixtures block matters more than people expect at this stage: with no production data, your tool responses are invented, so pin them. Recorded or stubbed tool responses are what keeps a synthetic case from flaking for reasons that have nothing to do with the agent.
Step 1: enumerate from the tool schemas
Your agent's tool definitions are a specification of everything it can do. Read them mechanically before you write a single sentence of English.
For each tool, enumerate:
- The happy path. One case where this tool is the correct call.
- The near-miss. One case that looks like this tool's job but is actually another tool's, or none. If
issue_refundandcancel_orderboth plausibly answer "I don't want this anymore", you need a case for each. - The missing-argument case. A request that leaves a required parameter unstated. The agent should ask, not invent. This is the abstention boundary.
- The out-of-scope case. A request the tool set genuinely cannot serve. Correct behaviour is to say so.
A five-tool agent gives you twenty cases before any creativity is involved. Write the enumeration as a script over the JSON schemas so the skeleton regenerates when the tool set changes:
import json
SHAPES = ["happy", "near_miss", "missing_arg", "out_of_scope"]
def skeletons(tools):
for tool in tools:
required = tool["inputSchema"].get("required", [])
for shape in SHAPES:
yield {
"id": f"{tool['name']}-{shape}",
"tool": tool["name"],
"shape": shape,
"required_args": required,
"input": None, # filled in step 2
"expect": None, # written by a human in step 4
}
tools = json.load(open("tools.json"))
skel = list(skeletons(tools))
print(len(skel), "skeletons")
Step 2: generate the wording with a model, not the expectations
Now the LLM step. Use it for exactly one job: turning a skeleton into user phrasing that sounds like a real person. Do not let it write the assertions. A model that invents both the question and the right answer is grading its own homework, and you will ship a dataset that encodes the model's assumptions about your product.
PROMPT = """You write realistic user messages for a support agent.
Product context:
{context}
Write {n} distinct user messages that would require the agent to use the
tool "{tool}" in the "{shape}" situation described below.
{shape_description}
Vary length, tone, typos and how much detail the user volunteers.
Some users are terse. Some ramble. Some are annoyed.
Return a JSON array of strings, nothing else."""
Three details make the output usable:
- Give it real product context — the tool descriptions, a couple of genuine support emails, your actual entity names. Generic prompts produce generic cases and generic cases pass.
- Ask for variation explicitly. Left alone, a model writes the same well-formed sentence fifty times with different nouns. Your real users do not.
- Generate more than you need. Ask for 4x, then cut in step 3.
Promptfoo can do a version of this for you through its dataset generation, and Langfuse will hold the result as a versioned dataset. Either is fine. The method is what matters, not the runner.
Step 3: deduplicate, because the model repeats itself
Synthetic sets are always more redundant than they look. Fifty cases that are four cases in trenchcoats will give you a pass rate with a very confident, very wrong confidence interval. Embed and cull:
import numpy as np
def dedupe(texts, embed, threshold=0.92):
kept, vecs = [], []
for t in texts:
v = embed(t)
v = v / np.linalg.norm(v)
if vecs and max(float(v @ k) for k in vecs) > threshold:
continue
kept.append(t)
vecs.append(v)
return kept
Then check coverage the other way: a histogram of cases per tool, per shape. Cold-start sets skew hard toward whatever the product team talks about most. If issue_refund has fourteen cases and escalate_to_human has one, that is a finding about your team, not your agent.
Step 4: a human writes every expectation
This is the step people skip and it is the step that makes the set worth anything. For each surviving case, a person who knows the product decides what the agent must do, must not do, and what the tools return. It takes about three minutes a case. Fifty cases is an afternoon.
While doing it, expect to find that roughly one case in six has no agreed answer. Should the agent refund a delivered-but-damaged item under $25 without a photo? Two people on the team will say different things. That disagreement is the most valuable output of the whole exercise: it is a product decision that was never made, surfacing before a customer finds it instead of after. Log those separately and get a ruling.
Cases whose correct answer is genuinely fuzzy get a rubric and a judge — and a judge with no measured agreement against human labels is a guess, so calibrate it on these same cases before you believe a number.
Step 5: label the set for what it is, and plan its replacement
Tag every case with its provenance:
provenance: synthetic-cold-start
created: 2026-09-14
reviewed_by: dana
You want that tag for two reasons. First, when the set starts passing at 96%, provenance tells you whether that is good news or evidence you wrote an easy test and then optimized against it. Second, it lets you track the ratio that actually matters after launch: synthetic cases versus cases mined from real traces. Watch it fall. In a healthy suite, most synthetic cases are retired or rewritten within a couple of months of live traffic, because real users do things nobody on your team thought to invent.
What this set can and cannot tell you
It can tell you the agent handles the paths you can describe, respects tool ordering and budgets, asks instead of guessing when an argument is missing, and refuses what it cannot serve. That is a real answer to "is anything obviously broken", and it is enough to catch regressions from day one of CI.
It cannot tell you your production pass rate. Nothing built before launch can. It will miss the distribution of real traffic, the weird inputs, and the failure modes that only appear at volume. Anyone who tells you a synthetic suite predicts production numbers is selling something.
What it buys you is a defensible launch decision and a harness that is ready the moment real traces start arriving. That starter set — plus a trace review, a failure taxonomy and a ranked go/no-go — is what an agent readiness assessment delivers in about two weeks. The method above is the same one we use; if you have an afternoon and someone who knows the product, run it yourself.