A team told us their agent eval suite was "about 80% reliable". Same commit, same dataset, pass rate between 71% and 88% run to run. The model was part of it. Most of it was the tools: a search API that reranked its results, a CRM sandbox other people were writing to, a weather tool that returned a different day, and one endpoint that timed out about one call in forty.
An eval suite that calls live tools measures your vendors' Tuesday as much as it measures your agent. This tutorial fixes the tool half: record real tool responses once, replay them from disk on every run, and keep a separate nightly job that checks the recordings still match reality. The model half — sampling noise — is a different problem, covered in measuring noise in agent pass rates.
What determinism buys you
With replayed tools, a red diff means the agent changed. That is the whole point. Three concrete wins:
- Bisectable failures. A failing case fails the same way on your laptop, in CI, and next quarter.
- Cheap and fast runs. No rate limits, no sandbox seeding, no bill. Two hundred cases run in the time twenty used to.
- Safe adversarial cases. You can record a hostile payload once and replay it forever without poking a live system.
And one thing it does not buy you: proof the agent works against the current API. That is what step 4 is for.
Step 1: put a seam in front of your tools
You need one function every tool call passes through. If you already have a tool dispatcher, you have the seam. If your agent calls requests in eight places, add one.
# tools.py
def call_tool(name: str, arguments: dict) -> dict:
"""Single entry point for every tool the agent can invoke."""
return REGISTRY[name](**arguments)
Everything below wraps this one function. Doing it at the HTTP layer (VCR-style) also works and catches more, but tool-level recording is easier to read in review and survives a client library upgrade.
Step 2: record cassettes
A cassette is a JSON file per eval case holding the tool interactions of one run, in order:
{
"case_id": "refund-damaged-item",
"recorded_at": "2026-09-02T14:11:03Z",
"interactions": [
{
"tool": "lookup_order",
"arguments": {"order_id": "4832"},
"response": {"order_id": "4832", "status": "delivered", "total_cents": 4200}
},
{
"tool": "issue_refund",
"arguments": {"order_id": "4832", "amount_cents": 4200},
"response": {"refund_id": "rf_99", "status": "ok"}
}
]
}
The recorder is a wrapper around the seam:
# cassette.py
import json, os, datetime
class Recorder:
def __init__(self, case_id, path):
self.case_id, self.path, self.interactions = case_id, path, []
def call_tool(self, name, arguments):
response = real_call_tool(name, arguments)
self.interactions.append(
{"tool": name, "arguments": arguments, "response": response}
)
return response
def save(self):
with open(self.path, "w") as f:
json.dump(
{
"case_id": self.case_id,
"recorded_at": datetime.datetime.utcnow().isoformat() + "Z",
"interactions": self.interactions,
},
f,
indent=2,
sort_keys=True,
)
Three rules while recording, learned the hard way:
- Record against a sandbox, never production. If a case can send an email, delete a record or move money, fix that before you record it.
- Redact at write time, not at review time. Run responses through a scrubber for tokens, emails, names and account ids before they hit disk. A cassette is a fixture that lives in your repo forever.
- Commit cassettes with the case, and re-record deliberately.
RECORD=1 pytest -k refundregenerates one case. A pull request that re-records forty cassettes at once deserves a hard look.
Step 3: replay in pytest
Replay looks up the recorded response by tool name plus a normalized argument key, so it does not depend on call order:
class Player:
def __init__(self, cassette):
self.index = {}
for i in cassette["interactions"]:
self.index.setdefault(self._key(i["tool"], i["arguments"]), []).append(i["response"])
self.unmatched = []
@staticmethod
def _key(name, arguments):
return name, json.dumps(arguments, sort_keys=True)
def call_tool(self, name, arguments):
queue = self.index.get(self._key(name, arguments))
if not queue:
# The agent asked for something the recording does not contain.
self.unmatched.append({"tool": name, "arguments": arguments})
raise CassetteMiss(f"no recorded response for {name}({arguments})")
return queue.pop(0) if len(queue) > 1 else queue[0]
Wire it up as a fixture:
# conftest.py
import json, os, pytest
@pytest.fixture
def tools(request, monkeypatch):
case_id = request.node.callspec.params["case"]["id"]
path = f"cassettes/{case_id}.json"
if os.environ.get("RECORD"):
driver = Recorder(case_id, path)
else:
driver = Player(json.load(open(path)))
monkeypatch.setattr("tools.call_tool", driver.call_tool)
yield driver
if os.environ.get("RECORD"):
driver.save()
And the eval itself asserts on the trajectory, the same structural checks you would write with Promptfoo:
@pytest.mark.parametrize("case", load_cases(), ids=lambda c: c["id"])
def test_agent_case(case, tools):
result = run_agent(case["input"])
names = [c["name"] for c in result.tool_calls]
assert "lookup_order" in names
assert names.index("lookup_order") < names.index("issue_refund")
assert names.count("issue_refund") == 1
assert tools.unmatched == []
Cassette misses are signal, not plumbing
The instinct when a replay misses is to loosen matching until it passes. Resist it. A miss usually means one of three things, and all three are worth knowing:
- The agent called a different tool or different arguments than when you recorded. That is a behavior change — the exact thing the suite exists to catch.
- Your arguments contain incidental variance: a timestamp, a UUID, a floating-point total. Normalize those in
_keyexplicitly, one field at a time, and write down why. - The case genuinely needs a new tool interaction. Re-record that one case, and read the diff.
Match strictly by default. Every loosening is a hole in the assertion.
Nondeterministic-by-nature tools
Some tools are supposed to vary: a search index that updates hourly, a pricing endpoint. Replay pins them to a recorded snapshot, which is exactly what you want for regression testing. Just be honest about what the suite then measures: agent behavior given that snapshot. If the agent's job includes coping with results that shift, record two or three cassette variants for the same input and assert the invariant that must hold across all of them.
Step 4: the nightly contract check
Replayed fixtures rot. The vendor renames a field, adds a required parameter, changes an enum, and your suite goes on happily passing against a fossil. So run a second, small job — nightly or weekly, not on every pull request — that calls the real tools in a sandbox and compares the shape of what comes back to the recordings:
def test_cassette_contract(interaction):
live = real_call_tool(interaction["tool"], interaction["arguments"])
assert schema_of(live) == schema_of(interaction["response"]), (
f"{interaction['tool']} response shape drifted; re-record"
)
Compare types and key sets, not values — values are supposed to differ. When this job fails, the fix is to re-record and read the diff, because a changed tool schema is often a real agent bug waiting to happen. Sample it if you have hundreds of cassettes: one interaction per distinct tool per night is enough to catch drift early.
Two suites, two jobs: the fast deterministic one gates merges, the slow live one tells you when the fixtures are lying. Keep them separate so nobody is tempted to make the merge gate flaky again.
What this does not cover
Deterministic tools remove one source of variance. What is left:
- Model sampling. Even at temperature 0, providers do not guarantee identical output. Run repeats and read pass rates as distributions, not lights.
- Judge variance. Any LLM-graded assertion adds its own noise on top, and needs calibration against human labels before you trust it.
- Reality. A suite that only replays cassettes eventually tests a museum. Feed it from live traffic — see building an eval dataset from production traces.
We build this harness in most eval suite engineering engagements, and it lands as code in the client's repository, cassettes and all. It is not glamorous work. It is the difference between an eval suite people trust and one they re-run until it goes green.