Splitting one agent into a router plus specialists is a reasonable move. The single-agent prompt gets long, tool lists get confusing, and handing "billing questions" to a billing agent with four tools beats giving one agent forty. Frameworks make it easy: the OpenAI Agents SDK exposes handoffs as tools, LangGraph does it as edges between nodes, CrewAI as delegation.
What nobody hands you is the eval suite. Splitting the agent does not remove failure modes, it adds three:
- Routing errors. The request goes to the wrong specialist, or stays with the router.
- Context loss. The specialist gets the handoff but not the order id, the tenant id, or the fact that the customer already tried a restart.
- Ping-pong. Two agents hand the same task back and forth until the step budget runs out.
Your existing end-to-end evals catch some of this indirectly: the final answer is wrong, the case fails, and you spend an hour reading a trace to find out why. Handoff-level evals tell you which of the three it was, in the failure message. This tutorial builds them in pytest against any framework, because the assertions read the trajectory, not the framework internals.
Step 1: get the handoffs out as data
Every one of these checks needs the same record: who handed off to whom, when, and with what payload. Frameworks disagree on where that lives, so normalize once and write all your assertions against the normalized shape.
# handoffs.py
from dataclasses import dataclass, field
@dataclass
class Handoff:
source: str # agent that gave up the turn
target: str # agent that received it
payload: dict = field(default_factory=dict) # what was passed along
step: int = 0 # position in the run
@dataclass
class Run:
entry_agent: str
handoffs: list[Handoff]
final_agent: str
answer: str
tool_calls: list[dict] = field(default_factory=list)
In the OpenAI Agents SDK, handoffs appear in result.new_items as handoff items, and the structured input is whatever your input_type model captured. In LangGraph, read the node transitions off the state stream. If you already emit OpenTelemetry GenAI spans, each agent is a span and a handoff is a parent-child edge with the agent name attribute; that path is the most portable one, because it survives a framework change.
Whatever the source, the adapter is small and it is the only framework-specific code in the suite:
def to_run(result) -> Run:
handoffs = [
Handoff(
source=item.source_agent.name,
target=item.target_agent.name,
payload=getattr(item, "input", {}) or {},
step=i,
)
for i, item in enumerate(result.new_items)
if item.type == "handoff_output_item"
]
return Run(
entry_agent=result.entry_agent_name,
handoffs=handoffs,
final_agent=handoffs[-1].target if handoffs else result.entry_agent_name,
answer=result.final_output,
tool_calls=extract_tool_calls(result),
)
Step 2: score routing as a classification problem
Routing is a classifier. Score it like one. Write the cases as data with the agent you expected to end up on, run them, and build a confusion matrix rather than a single pass rate — the matrix is what tells you which pair of specialists your router cannot tell apart.
# cases/routing.yaml
- id: refund_damaged
input: "My blender arrived cracked, I want my money back"
expect_agent: billing
- id: password_reset
input: "Can't log in, reset link never arrives"
expect_agent: account
- id: mixed_billing_first
input: "I was charged twice and also can't log in"
expect_agent: billing # policy: money issues win
- id: out_of_scope
input: "What do you think of the new pricing at your competitor?"
expect_agent: triage # stays put, does not hand off
# test_routing.py
import pytest, yaml
from collections import Counter
from handoffs import to_run
CASES = yaml.safe_load(open("cases/routing.yaml"))
CONFUSION = Counter()
@pytest.mark.parametrize("case", CASES, ids=lambda c: c["id"])
def test_routes_to_expected_agent(case, agent_app):
run = to_run(agent_app.invoke(case["input"]))
CONFUSION[(case["expect_agent"], run.final_agent)] += 1
assert run.final_agent == case["expect_agent"], (
f"{case['id']}: routed to {run.final_agent}, "
f"path {[h.target for h in run.handoffs]}"
)
Two details matter more than the code.
The out_of_scope case is not optional. Routers over-hand-off: given a specialist named billing, they will send it anything containing a currency symbol. Half your routing cases should be ones where the correct behaviour is to answer directly or to say no. Without them you measure recall and stay blind to precision.
Write down the tie-break policy before you write the cases. mixed_billing_first only has a right answer because someone decided money issues outrank login issues. If you cannot state the policy, the eval is not measuring the agent, it is measuring your mood on labelling day. Put the rule in the router prompt and in the case file comment, in the same words.
Print the matrix at the end of the run and look at the off-diagonal:
def pytest_sessionfinish(session, exitstatus):
for (expected, actual), n in sorted(CONFUSION.items()):
if expected != actual:
print(f"MISROUTE {expected} -> {actual}: {n}")
Clusters on one cell mean a prompt fix (the two specialist descriptions overlap). Misroutes scattered everywhere mean the split itself is wrong, and merging two specialists will beat any amount of prompt tuning.
Step 3: assert that context survived the handoff
Routing to the right agent and then losing the order id still fails the user; it usually shows up as the specialist re-asking a question the customer already answered. This is the failure mode teams most often miss, because the trace looks correct until you read the payload.
Check the payload directly. It is structured data, so no judge is needed:
REQUIRED_BY_TARGET = {
"billing": {"tenant_id", "order_id"},
"account": {"tenant_id", "user_id"},
}
def test_handoff_payload_complete(agent_app):
run = to_run(agent_app.invoke(
"Order 4832 arrived cracked, refund it — account jane@acme.test"
))
for h in run.handoffs:
required = REQUIRED_BY_TARGET.get(h.target, set())
missing = required - set(k for k, v in h.payload.items() if v)
assert not missing, f"{h.source}->{h.target} dropped {missing}"
Two more checks belong here:
- No re-asking. Assert the specialist's first message does not ask for a fact already present in the payload. A regex over the question mark plus the field name gets you most of the way; keep it deterministic before reaching for a rubric.
- No invention. If
order_idis absent from the user's message, assert it is absent from the payload too. Routers under pressure to fill a schema will make up an id, and a fabricated id that happens to exist in the database is the worst outcome in this whole tutorial. If the payload carries atenant_id, the same probe belongs in your authorization suite — a specialist should never receive a tenant it was not given.
Step 4: catch ping-pong before it bills you
Two agents each convinced the other owns the task will trade it until something stops them. Cap it explicitly rather than relying on the framework's default:
def test_no_handoff_loops(agent_app):
run = to_run(agent_app.invoke("I want to talk to a person about my invoice"))
edges = [(h.source, h.target) for h in run.handoffs]
assert len(edges) <= 3, f"handoff budget exceeded: {edges}"
assert len(edges) == len(set(edges)), f"repeated handoff edge: {edges}"
# A -> B -> A is the classic shape
for i in range(len(edges) - 1):
assert edges[i][0] != edges[i + 1][1], f"ping-pong at step {i}: {edges}"
Escalation and "talk to a human" requests are the reliable trigger, so seed those cases deliberately. The same budget thinking applies to steps and tokens across the whole run; that is budget evals, and multi-agent runs are where the numbers get ugly, because each handoff usually replays context into a fresh model call.
Step 5: run it in CI, and track the split
pytest test_routing.py test_handoffs.py -q --junitxml=results.xml
Report three numbers per run, not one:
| Metric | What it answers | Fails when |
|---|---|---|
| Routing accuracy | did the request reach the right specialist | prompts or agent descriptions overlap |
| Payload completeness | did the facts travel with it | handoff schema or extraction is wrong |
| Handoff budget (max edges) | did the agents argue | ownership rules are ambiguous |
Keeping them separate is the point. A single end-to-end pass rate that drops from 0.82 to 0.71 after a prompt change tells you to go read traces. Three metrics tell you routing held, payloads held, and the loop check broke on escalation cases — which is a fifteen-minute fix.
Gate the merge on the routing suite the same way you gate any other eval, with per-case flake measured first so you are not chasing noise. Both of those are covered in the CI gating and pass-rate noise tutorials.
What this does not tell you
These checks say the right specialist got the work with the right facts and without arguing. They say nothing about whether the specialist then did the job correctly — that is ordinary trajectory and tool-call evaluation, per agent, and you still need it. Nor do they replace red-teaming: a specialist that trusts its handoff payload as instructions is a confused-deputy problem, and confirming that requires adversarial cases, not routing cases.
One more thing worth saying plainly. If routing accuracy on a well-labelled set sits below the high eighties and the misroutes are spread across pairs, the honest finding is usually that the architecture is over-split. We have handed that finding to clients more than once: merging two specialists back into one agent removed more failures than any prompt work would have. Measure first, then decide — the measurement is cheap compared to the debugging it replaces.
If you want a second pair of eyes on a multi-agent system before launch, get in touch and describe the agents, the tools and the timeline. An engineer replies within one business day.