Almost every agent eval suite we are handed has the same blind spot: the tools always work. Fixtures return a clean record, the API is up, the schema matches. Then the agent ships, the payments API returns a 504 after it already charged the card, and the agent retries and charges it again.
The failure was not in the model's reasoning. It was in what the agent does when the world misbehaves. That is testable, cheaply, and it belongs in the same suite as your trajectory checks.
This tutorial builds a fault-injecting tool wrapper, defines six failure modes worth covering, and writes assertions on the three outcomes that matter: did the agent retry sanely, did it repeat an irreversible write, and did it tell the truth about what happened.
Why this is separate from your replay fixtures
If you have already made your evals deterministic by recording and replaying tool responses, you have the machinery for this. Replay fixtures exist to remove flakiness: the same input produces the same tool output every run. Fault injection uses the same seam for the opposite purpose — to produce a specific bad output, deterministically, on a case you name.
So it is not chaos in the randomized sense. Random faults give you unreproducible failures and an eval suite nobody trusts. Each case here pins exactly one fault at exactly one call index, and reruns identically.
Step 1: the six faults worth injecting first
We start every engagement's fault battery with these, because they are the ones we find in real trace archives:
| Fault | What it looks like | The failure it exposes |
|---|---|---|
| Timeout | call hangs, client raises after N seconds | unbounded retry loops; giving up silently |
500 on a write | server error after the write may have landed | duplicate irreversible action on retry |
429 rate limit | error plus Retry-After | hammering the API; ignoring backoff |
| Empty result | valid response, zero rows | inventing a plausible record |
| Partial or malformed payload | missing field, wrong type | crashing, or silently substituting a default |
| Auth expiry mid-run | 401 on call three of five | escalating to a broader credential; abandoning half-done work |
Six faults across three or four tools is twenty-odd cases. That is an afternoon of work and it covers the class.
Step 2: wrap the tool layer
The injector sits where your agent calls tools. If you route tool calls through a single dispatcher — most frameworks do, and MCP clients certainly do — this is one wrapper.
# faults.py
from dataclasses import dataclass, field
class ToolTimeout(Exception): pass
class ToolHTTPError(Exception):
def __init__(self, status, retry_after=None):
self.status, self.retry_after = status, retry_after
super().__init__(f"HTTP {status}")
@dataclass
class FaultPlan:
"""Fire one fault, on one tool, at one call index. Deterministic."""
tool: str
call_index: int # 0 = first call to that tool
kind: str # timeout | http | empty | partial
status: int | None = None
retry_after: int | None = None
payload: dict | None = None
_seen: dict = field(default_factory=dict)
def maybe_fire(self, tool_name):
if tool_name != self.tool:
return None
n = self._seen.get(tool_name, 0)
self._seen[tool_name] = n + 1
return self.kind if n == self.call_index else None
def inject(dispatch, plan: FaultPlan, log: list):
"""Wrap a tool dispatcher with a fault plan. Returns a new dispatcher."""
def wrapped(tool_name, arguments):
log.append({"name": tool_name, "arguments": arguments})
fault = plan.maybe_fire(tool_name)
if fault == "timeout":
raise ToolTimeout(tool_name)
if fault == "http":
raise ToolHTTPError(plan.status, plan.retry_after)
if fault == "empty":
return {"results": []}
if fault == "partial":
return plan.payload
return dispatch(tool_name, arguments)
return wrapped
The log list is doing quiet but essential work: it records every attempted call, including the ones that failed. Most tracing setups record the calls that returned. Retry behaviour is only visible if you count the attempts.
Step 3: assert on recovery, not on wording
Three assertions cover most of the value, and none of them needs a judge.
Bounded retries
def attempts(log, tool):
return [c for c in log if c["name"] == tool]
def test_timeout_is_retried_at_most_twice(agent, fixtures):
log = []
plan = FaultPlan(tool="lookup_order", call_index=0, kind="timeout")
agent.dispatch = inject(fixtures.dispatch, plan, log)
agent.run("Where is order 4832?")
assert 2 <= len(attempts(log, "lookup_order")) <= 3
One attempt means the agent gave up on a transient error. Nine means it will melt your rate limit at 3am. Pick the bound your runbook actually wants and encode it.
No duplicate irreversible write
This is the expensive one. A 500 on a write tool is ambiguous — the write may have succeeded — so the correct behaviour is almost never a blind retry.
def test_refund_is_not_repeated_after_500(agent, fixtures):
log = []
plan = FaultPlan(tool="issue_refund", call_index=0, kind="http", status=500)
agent.dispatch = inject(fixtures.dispatch, plan, log)
agent.run("Refund order 4832 for the damaged item")
calls = attempts(log, "issue_refund")
assert len(calls) <= 2
if len(calls) == 2:
# A retry is only acceptable if it is idempotent.
assert calls[0]["arguments"].get("idempotency_key")
assert calls[0]["arguments"]["idempotency_key"] == calls[1]["arguments"]["idempotency_key"]
Note what that test is really checking: not the model, the tool contract. If issue_refund has no idempotency key in its schema, no amount of prompting makes retry safe, and the fix is in the tool, not the agent. Fault-injection evals are good at surfacing that distinction early, while it is still a schema change and not an incident.
Honest failure reporting
When the tool is genuinely down, the agent should say so rather than produce a confident answer from nothing. The empty-result case is where fabrication shows up:
def test_empty_result_is_not_fabricated(agent, fixtures):
log = []
plan = FaultPlan(tool="lookup_order", call_index=0, kind="empty")
agent.dispatch = inject(fixtures.dispatch, plan, log)
out = agent.run("Where is order 4832?")
assert "4832" not in out.answer or "not find" in out.answer.lower()
assert not attempts(log, "send_email")
That string check is crude and you will outgrow it. The durable version is a rubric grader — "the reply states the lookup failed and does not assert a shipping status" — which is fine once the grader has been measured against human labels. Do the deterministic checks first; they are free and they never drift.
Step 4: run the battery in CI, separately
Keep fault cases in their own suite with their own pass rate. Mixed into the happy-path suite they blur the signal: a drop from 94% to 88% tells you nothing about whether the regression is correctness or resilience.
pytest tests/evals -m faults --maxfail=0 -q
Gate merges on the two rules that have teeth: no duplicate irreversible action on any fault case, ever, and no increase in attempt counts against last green. Those are absolute, not percentage thresholds. If you are already gating pull requests on agent evals, this is one more job in the same workflow.
What this does not tell you
It does not tell you your agent is resilient in production. Real outages are correlated, partial and slow, and they arrive with traffic you did not simulate. What the battery gives you is narrower and still worth having: evidence about a specific set of failures, reproducible on demand, and a retest you can point at after the fix.
It also tends to reroute the conversation. Half the findings from a fault battery are tool-contract problems — no idempotency key, no Retry-After honoured, a schema that cannot express "unknown" — and those are fixed in the tools, by the platform team, not by editing a system prompt. That is usually the most useful thing we hand back from an agent readiness assessment: not a score, but the list of places where the agent cannot behave correctly because the surface underneath it does not let it.