A deprecation email arrives: the model your agent runs on retires in six weeks. Or a new version lands and someone in the standup says it is better at tool use. Either way you are being asked to change the most load-bearing component in the system, and the usual evidence is a benchmark chart from the vendor and a few prompts somebody tried by hand.
A benchmark tells you how a model does on someone else's task. It says nothing about whether your agent still calls issue_refund exactly once. This tutorial builds the smallest harness that answers that question: run the same cases against both models, paired, and diff what the agent did.
What actually breaks in a model swap
From migrations we have run and reviewed, the failures cluster:
| Change | What it looks like in traces |
|---|---|
| Tool-call formatting | Arguments arrive as a JSON string instead of an object; enum values come back capitalized differently |
| Step count | The new model plans more, calls the search tool three times where the old one called it once |
| Refusal boundary | Cases that used to proceed now return a policy refusal, or the reverse |
| Verbosity | Longer final answers, more tokens, higher p95 latency, higher bill |
| Instruction adherence | The system prompt tuned against the old model's quirks stops landing |
None of these show up as "quality dropped". They show up as a customer-visible bug three days after the swap. All of them are structural, which means you can test for them without a judge model.
Step 1: freeze the inputs
A migration eval is only meaningful if the model is the sole variable. That means the same cases, the same prompts, the same tool responses. Live tools break this immediately — the search index moves, the order record changes — so replay them from recorded fixtures. If you do not have that yet, the tool-replay cassette harness is the prerequisite, and it is an afternoon of work.
Cases come from production traffic, not imagination. Pull a stratified sample of real sessions the way described in building an eval dataset from production traces. Sixty cases is enough to see the big shifts; two hundred is enough to argue about small ones.
# cases.py
import json
from pathlib import Path
def load_cases():
for path in sorted(Path("cases").glob("*.json")):
yield json.loads(path.read_text())
Each case file is the user input, the cassette id for its tool responses, and whatever expectations you already labeled:
{
"id": "refund-damaged-item",
"input": "Refund order 4832, the item arrived damaged",
"cassette": "refund-damaged-item",
"must_call": ["lookup_order", "issue_refund"],
"max_steps": 8
}
Step 2: run both models over the same cases
The run function returns the trajectory, not just the answer. Everything downstream reads it.
# migration_run.py
import json, time
from cases import load_cases
from myagent import build_agent
from replay import cassette
MODELS = ["gpt-4.1-2025-04-14", "gpt-5.1-2026-02-03"] # old, new
REPEATS = 3
def run_one(model, case):
agent = build_agent(model=model, temperature=0)
started = time.perf_counter()
with cassette(case["cassette"]):
result = agent.run(case["input"])
return {
"case": case["id"],
"model": model,
"answer": result.answer,
"tool_calls": [
{"name": c.name, "arguments": c.arguments} for c in result.tool_calls
],
"refused": result.stop_reason == "refusal",
"input_tokens": result.usage.input_tokens,
"output_tokens": result.usage.output_tokens,
"latency_s": round(time.perf_counter() - started, 3),
}
if __name__ == "__main__":
with open("runs.jsonl", "w") as out:
for case in load_cases():
for model in MODELS:
for i in range(REPEATS):
row = run_one(model, case)
row["repeat"] = i
out.write(json.dumps(row) + "\n")
Three repeats per case per model is not superstition. Agents are nondeterministic even at temperature zero, and without repeats you cannot tell a model difference from a flake. The pass-rate noise tutorial covers how many runs you need before a delta means anything; if you skip it, at minimum report how often each model gave the same trajectory across its own repeats.
One run of sixty cases against two models with three repeats is 360 agent runs. On replayed tools that is minutes, not hours, and the only cost is model tokens.
Step 3: diff the trajectories, case by case
Aggregate pass rates hide the thing you need to see. What matters is the list of cases that changed and in which direction.
# migration_report.py
import json
from collections import defaultdict
OLD, NEW = "gpt-4.1-2025-04-14", "gpt-5.1-2026-02-03"
def passes(row, case):
names = [c["name"] for c in row["tool_calls"]]
return (
all(t in names for t in case["must_call"])
and len(row["tool_calls"]) <= case["max_steps"]
and not row["refused"]
)
def signature(row):
return "|".join(c["name"] for c in row["tool_calls"])
rows = [json.loads(l) for l in open("runs.jsonl")]
cases = {c["id"]: c for c in __import__("cases").load_cases()}
by = defaultdict(list)
for r in rows:
by[(r["case"], r["model"])].append(r)
fixed, broken, reshaped = [], [], []
for cid, case in cases.items():
old_ok = all(passes(r, case) for r in by[(cid, OLD)])
new_ok = all(passes(r, case) for r in by[(cid, NEW)])
if new_ok and not old_ok:
fixed.append(cid)
if old_ok and not new_ok:
broken.append(cid)
old_sigs = {signature(r) for r in by[(cid, OLD)]}
new_sigs = {signature(r) for r in by[(cid, NEW)]}
if old_ok and new_ok and not (old_sigs & new_sigs):
reshaped.append(cid)
print("broken :", broken)
print("fixed :", fixed)
print("same result, different route:", reshaped)
That third list is the one teams forget. A case can still pass while the agent takes a completely different path to get there — two extra searches, a different write tool, a fallback branch nobody has read in months. It passed today. It is a different system tomorrow.
Print the broken list with the actual trajectories side by side. In every migration review we have sat in, the argument stops the moment someone reads two call sequences next to each other.
Step 4: check the boring deltas
Correctness is not the whole decision. Four numbers belong in the same report:
def agg(model, key):
vals = [r[key] for r in rows if r["model"] == model]
vals.sort()
return {
"mean": round(sum(vals) / len(vals), 3),
"p95": vals[int(len(vals) * 0.95) - 1],
}
for key in ("latency_s", "output_tokens"):
print(key, OLD, agg(OLD, key), "->", NEW, agg(NEW, key))
for model in (OLD, NEW):
refusals = sum(r["refused"] for r in rows if r["model"] == model)
print(model, "refusals", refusals, "/", sum(r["model"] == model for r in rows))
for model in (OLD, NEW):
steps = [len(r["tool_calls"]) for r in rows if r["model"] == model]
print(model, "mean steps", round(sum(steps) / len(steps), 2), "max", max(steps))
Report p95 latency, not the mean — the mean hides the tail your users complain about. Report refusal counts in both directions; a model that refuses less is not automatically better, and a model that refuses more will generate support tickets. Report mean and max steps, because more steps is more tokens, more tool load and more chances to go wrong. Budget assertions belong in this report too; the mechanics are in budget evals.
Step 5: keep the schema honest
Tool-call formatting is where migrations quietly break. Validate every call the new model makes against your tool schema and fail loudly:
import jsonschema, pytest
from tools import SCHEMAS
@pytest.mark.parametrize("row", [r for r in rows if r["model"] == NEW], ids=lambda r: r["case"])
def test_tool_arguments_match_schema(row):
for call in row["tool_calls"]:
jsonschema.validate(call["arguments"], SCHEMAS[call["name"]])
This catches arguments arriving as a JSON string, missing required fields, enum drift and silently coerced types — the class of failure that a pass-rate number never surfaces because your tool layer forgives it in staging and the production API does not.
Step 6: write down the decision
The output of a migration eval is not a green check. It is a short document: cases broken, cases fixed, trajectory changes on passing cases, latency and token deltas, refusal deltas, and what you are doing about each broken case. Then a plan — canary a slice of traffic, watch the same scores on live traces the way Langfuse scoring describes, and keep the old model reachable behind a config flag until the canary is boring.
Add the migration run to CI while you are there. A model version is a dependency; pinning it and re-running the suite when it changes is the same discipline as pinning a library. The CI gating tutorial has the wiring.
What this does not tell you
It does not tell you the new model is safer. Adversarial behaviour changes across models too — injection susceptibility, authorization boundaries, exfiltration paths — and none of that is measured by replaying well-behaved production cases. Re-run the red-team battery against the new model before launch; that is a separate exercise and it is what agent red-teaming covers.
It also does not tell you whether the final answers are better in the ways a human cares about. If you want that, add a rubric grader and measure the grader first — an uncalibrated judge will happily report whichever model wrote longer answers.
What it does give you, in a day of work on cases you already have, is the list of things that changed. That is the honest basis for a migration decision, and it is one you can rerun the next time a deprecation email arrives.
If a deprecation deadline is closer than your eval suite is finished, tell us about the agent, its tools and the date. An engineer replies within one business day with what we would test first.