Agents accumulate tools. You start with four, connect two MCP servers, and six months later the model is choosing from ninety function definitions, half of which it has never needed. Then the pass rate on your old eval set stays green while real users get the wrong tool, the deprecated tool, or three tools when one would do.
This is a selection problem, not a reasoning problem, and it has its own tests. This tutorial builds them: a confusion matrix over tool choice, an ablation that tells you which definitions are earning their context, and regression cases for the day somebody adds a server with a search tool to a catalog that already has one.
Why a big catalog degrades quietly
Three mechanisms, all boring:
| Mechanism | What you see in traces |
|---|---|
| Near-duplicate names and descriptions | Agent picks search_docs when search_tickets was right; both say "search for relevant information" |
| Description drift | A vendor's MCP server updates a description, selection shifts, nothing in your repo changed |
| Context dilution | Ninety schemas push the actual user request far from the instructions; the agent calls nothing, or calls the first plausible thing |
None of these produce an exception. The agent returns a fluent answer built on the wrong data source. Your end-to-end pass rate moves a couple of points and you blame the model version.
A related failure is the catalog changing under you without a code change. That is a contract-test problem and it has its own tutorial: contract tests for tool definitions. Here we assume the schemas are stable and ask a different question: given this catalog, does the agent pick correctly?
Step 1: label the ground truth per case
You need cases where you know which tool should fire. Pull them from traces you have already reviewed - building an agent eval dataset from production traces covers the mining - and label each one with the expected first tool call, not the whole trajectory. Keep it narrow on purpose: this suite answers "did it reach for the right thing," and the trajectory suite answers the rest.
{"id": "sel-001", "input": "what did we tell Acme about the SLA credit", "expected_tool": "search_tickets"}
{"id": "sel-002", "input": "how do I rotate an API key", "expected_tool": "search_docs"}
{"id": "sel-003", "input": "thanks, that's all", "expected_tool": null}
{"id": "sel-004", "input": "cancel Acme's renewal", "expected_tool": "lookup_account"}
The null cases matter more than the rest. An agent that always calls something is a worse agent, and a suite made only of positive cases will never tell you. Aim for 15-20% no-tool cases. Scoring whether the agent declines or asks is covered separately in escalation evals.
Thirty to sixty cases is enough to start. Cover every tool you actually care about with at least three cases, and deliberately write adjacent pairs - two inputs that differ only in the detail that should flip the choice.
Step 2: score it as a confusion matrix, not a pass rate
A single number tells you selection is 82% and nothing else. The matrix tells you that 82% is really "fine everywhere except search_docs is eating a third of search_tickets."
Run the agent with the full catalog, capture the first tool call, and tabulate:
# selection_eval.py
from collections import Counter
def run_matrix(cases, run_agent):
confusion = Counter()
for case in cases:
calls = run_agent(case["input"]) # your agent entry point
got = calls[0]["name"] if calls else None
confusion[(case["expected_tool"], got)] += 1
return confusion
def per_tool_report(confusion):
tools = {t for pair in confusion for t in pair if t}
for tool in sorted(tools):
tp = confusion[(tool, tool)]
fn = sum(n for (exp, got), n in confusion.items() if exp == tool and got != tool)
fp = sum(n for (exp, got), n in confusion.items() if got == tool and exp != tool)
recall = tp / (tp + fn) if tp + fn else float("nan")
precision = tp / (tp + fp) if tp + fp else float("nan")
print(f"{tool:24} precision={precision:.2f} recall={recall:.2f} fp={fp} fn={fn}")
Read the off-diagonal cells first. A tool with high recall and low precision is a magnet: its description is too broad and it is stealing calls from neighbours. A tool with low recall is invisible: nobody can tell from its description when it applies. Both are fixed in the description, not in the system prompt.
One caution on sampling. A selection rate measured once on sixty cases has a wide interval; three runs per case with a temperature above zero is the minimum before you believe a four-point move. Same arithmetic as measuring noise in agent pass rates.
Step 3: ablate the catalog to find what the tools cost
The interesting experiment is running the same cases with different catalogs. Three arms:
ARMS = {
"full": all_tools, # everything the agent sees in prod
"minimal": [t for t in all_tools if t["name"] in CORE], # the 8 you think matter
"minus_one": None, # full catalog less one tool, per tool
}
What you learn from each:
- full vs minimal. If minimal scores materially higher on the same cases, the catalog is costing you accuracy, and the fix is dynamic tool filtering - retrieve a shortlist of tools per request instead of pasting all ninety in.
- minus_one. Drop each tool in turn and re-run only the cases that do not need it. If accuracy on
search_ticketscases goes up whensearch_docsis absent, you have located the collision precisely. - full with rewritten descriptions. The cheapest intervention there is. Rewrite the two colliding descriptions to say when not to use the tool, re-run, and keep the change only if the matrix moves.
Record the arm name alongside every result so the comparison is a filter in your eval dashboard rather than a spreadsheet. In Langfuse this is a trace tag; in Promptfoo it is a provider per arm in the same config.
Step 4: gate the catalog, not just the prompt
Once the matrix is honest, wire two checks into CI next to your trajectory suite (gating a pull request on agent evals has the plumbing):
- Selection regression. Per-tool recall may not fall below the committed baseline by more than the noise band. Fail the build on the tool, and name it in the failure message.
- Catalog budget. Fail if the tool count crosses a threshold you chose on purpose. This is a forcing function: adding the ninety-first tool should require someone to look at the matrix.
Re-run the suite on three events, not just on code changes: a new MCP server connection, a model upgrade (selection behaviour is one of the first things that shifts - see model migration evals), and any vendor-side schema update your contract tests flag.
What this does not tell you
Selection accuracy is not correctness. An agent that picks the right tool with wrong arguments fails the user identically, and that is the trajectory suite's job (tool-call assertions in Promptfoo). A big catalog is also an authorization surface: more tools means more ways to reach data the caller should not see, which is a red-team question, not an accuracy one (authorization probes).
And this suite will not tell you whether to build tool retrieval. It will tell you what the full catalog costs you in accuracy. Weigh that against the latency and complexity of retrieving tools per request, and decide with the number in front of you.
If you want the selection matrix, the ablation arms and the CI gate built on your own stack and left in your repo, that is the middle of an eval suite engineering engagement. Tell us the catalog size and the tools that collide.