A team asked us to look at an agent whose refund pass rate fell from 92% to 78% over a weekend. No deploy. No model change. Same prompt, same eval suite, same commit SHA running in CI. The change was two words in somebody else's repository: an internal MCP server had edited the description of lookup_order from "Look up an order by ID" to "Look up an order by ID. Prefer search_orders when the ID is unknown." That sentence is now part of the agent's prompt, because tool definitions always are. The agent started reaching for search_orders first, burning a turn, and on long carts it ran out of its tool budget before it got to the refund.
This is the failure class nobody owns. You version your prompt. You pin your model. Most teams do not version the tool manifest their agent is handed at runtime, even though under MCP that manifest is fetched dynamically from servers other people deploy. tools/list is a network call. Anything reachable by a network call can change on a Tuesday.
The fix is boring and takes an afternoon: snapshot the manifest, diff it in CI, and treat each class of diff as a trigger for specific evals.
Step 1: snapshot the manifest your agent actually receives
Do not hand-write the snapshot from documentation. Capture what the runtime hands the model, after any filtering, renaming or namespacing your framework does. For an MCP client that is the result of tools/list per connected server; for plain function calling it is the tools array you pass to the API.
import hashlib
import json
def normalize(tools: list[dict]) -> list[dict]:
"""Stable, comparable view of the tool manifest."""
out = []
for t in sorted(tools, key=lambda t: t["name"]):
out.append(
{
"name": t["name"],
"description": (t.get("description") or "").strip(),
"input_schema": t.get("input_schema") or t.get("inputSchema") or {},
"annotations": t.get("annotations") or {},
}
)
return out
def fingerprint(tools: list[dict]) -> str:
blob = json.dumps(normalize(tools), sort_keys=True, separators=(",", ":"))
return hashlib.sha256(blob.encode()).hexdigest()[:16]
Write normalize(tools) to evals/fixtures/tool_manifest.json and commit it. Two details matter.
Sort by name. Several servers return tools in registration order, which changes when someone reorders a file. An unsorted snapshot produces a diff on every unrelated edit, and a diff that fires constantly gets ignored within a week.
Keep the description text verbatim, whitespace-stripped but otherwise untouched. The description is the part that changes agent behavior and the part everyone treats as documentation.
Step 2: fail the build on an undeclared change
The test is a snapshot comparison against a live manifest fetch.
import json
import pytest
FIXTURE = "evals/fixtures/tool_manifest.json"
@pytest.mark.asyncio
async def test_tool_manifest_matches_snapshot(mcp_client):
live = normalize(await mcp_client.list_tools())
with open(FIXTURE) as fh:
pinned = json.load(fh)
live_by_name = {t["name"]: t for t in live}
pinned_by_name = {t["name"]: t for t in pinned}
added = sorted(set(live_by_name) - set(pinned_by_name))
removed = sorted(set(pinned_by_name) - set(live_by_name))
changed = sorted(
n for n in set(live_by_name) & set(pinned_by_name)
if live_by_name[n] != pinned_by_name[n]
)
assert not (added or removed or changed), (
f"tool manifest drift\n added: {added}\n removed: {removed}\n changed: {changed}\n"
f"Re-run the affected evals, then update {FIXTURE} in the same PR."
)
Run it in two places, because it answers two different questions.
In CI on every pull request it asks: did this branch change the tools without updating the snapshot? That catches your own team.
On a schedule against staging — hourly is fine, it is one cheap call — it asks: did an upstream server change under us? That catches everyone else. Route the scheduled failure to whoever is on call for the agent, not to a channel nobody reads. In our experience this alert fires two or three times a quarter on any agent with more than a couple of internally-owned MCP servers, and roughly half of those firings are behavior-affecting.
One caveat: if a server gates its tool list by credential or feature flag, the manifest legitimately differs per environment. Snapshot per environment and name the fixture accordingly, rather than weakening the comparison to "names only." Names-only is the version that misses the two words that broke the refund flow.
Step 3: classify the diff, then run the right evals
A manifest diff is a signal, not a verdict. What you want is a triage table so the on-call engineer is not re-running the whole suite to investigate a typo fix.
| Diff | What it can break | Re-run |
|---|---|---|
| Tool removed or renamed | hard errors, agent improvises with a wrong tool | full trajectory suite; check fallback behavior |
| Required parameter added | every call missing it now fails | fault-injection cases, full suite |
| Required parameter relaxed to optional | agent silently stops passing it | cases that depend on that argument's effect |
| Enum value added or removed | invalid arguments, or new unreachable paths | cases touching that tool, plus authorization probes |
| Type or format change | coercion bugs at the boundary | tool-call assertion cases |
| Description or annotation edited | tool selection, ordering, turn count | full trajectory suite, tool-budget checks |
| New tool added | tool confusion, and a new attack surface | full suite plus the red-team battery |
Two rows deserve argument.
Description edits are not cosmetic. They are prompt edits made by a third party. Treat them with the same suspicion you would treat a stranger pushing to your system prompt, and re-run the trajectory suite. Selection and ordering assertions of the kind in regression-testing an agent's tool calls with Promptfoo are what actually detect this; a final-answer rubric usually will not, because the answer is often still correct — just slower and more expensive. Watch turn counts and cost alongside pass rate, per budget evals.
A new tool is a security change. It widens what the agent can be talked into doing, so the arrival of anything with write or network capability should pull in authorization probes and injection cases, not just correctness evals. If the new tool reads external content, it is a fresh injection vector and belongs in the battery described in testing your agent against poisoned MCP tool output.
Step 4: measure the behavior delta, do not guess it
When the diff is behavior-affecting, the useful artifact is a before/after run on the same dataset with both manifests.
If your tool responses are recorded fixtures, as in recording and replaying tool responses, you can do this in minutes: run the suite with the pinned manifest, run it with the live manifest, diff per case.
MANIFEST=evals/fixtures/tool_manifest.json pytest evals/ --json-report --json-report-file before.json
MANIFEST=live pytest evals/ --json-report --json-report-file after.json
python evals/compare_runs.py before.json after.json
Read it per case, not in aggregate. A pass rate that moves from 92% to 91% can hide six new failures and five new passes, and the six tell you something. What you are looking for:
- cases that flipped to failing, grouped by tool touched;
- median tool calls per case, which is where description edits show up first;
- any case where the agent called a tool it had never called before in the suite's history.
That last one is worth a standing check of its own. Keep the set of tool names your agent has ever emitted across the suite, and assert that a run introduces no new ones without a human acknowledging it.
And be honest about noise before you act. A two-point move on 60 cases is not evidence; the arithmetic is in measuring noise in agent pass rates. Run enough seeds that the delta you are attributing to the schema change is bigger than the delta you get from running twice with no change at all.
Step 5: make the pin part of the release record
Once you have a fingerprint, record it next to the other things you already pin. A release note that says "model gpt-x-2026-05, prompt v18, tools 9f2a1c4b7e03" lets you answer the weekend question — did anything change? — in one diff instead of an afternoon of archaeology.
Updating the snapshot is then a normal reviewed change: the PR contains the new fixture, the before/after eval numbers, and a sentence about why the change is acceptable. That is the whole discipline. The fixture is not the point; the required conversation is.
What this does not do
Contract tests on the manifest tell you the declaration changed. They say nothing about a server whose declaration is stable while its behavior is not — same schema, different data, slower responses, or a tool that starts returning content designed to steer your agent. Manifest pinning is not a defense against a compromised or hostile server, and we would not present it as one. For that you need adversarial cases against tool output, and the assumption that tool results are untrusted input. Nor does it help with a server that returns a different manifest to different callers; the pin only proves what you were handed.
What it does buy is the end of silent third-party prompt edits. When we run an Agent Readiness Assessment on an agent with more than three or four MCP servers wired in, an unpinned tool manifest is one of the most common findings, and it is close to the cheapest one to close.