Modern agent SDKs emit tool calls in parallel by default. One assistant turn returns four tool_use blocks, your runtime fires them together with asyncio.gather, and latency drops by 60%. It is the cheapest speedup available, so everyone takes it.
It also introduces a class of bug that single-threaded eval suites cannot see. Two calls that write the same record. A read that races the write that was supposed to precede it. A batch of five calls where the agent assumed an order the runtime never promised. These fail intermittently, pass on rerun, and look like flakes until someone reads the trace at 2am.
This tutorial builds a small deterministic concurrency harness and four evals on top of it: dependency ordering, read-after-write staleness, mutual exclusion on writes, and partial-batch failure. It is distinct from fault-injection evals, which ask what the agent does when a tool breaks. Here every tool works. The scheduling is what is hostile.
Why the normal harness misses this
Most eval harnesses execute tool calls one at a time in the order the model listed them. That is a fine default and it is also the single luckiest interleaving. Production runs whichever one the event loop happens to produce, and under load that is rarely the listed order.
So the harness has to do two things the production runtime does not: control the interleaving, and run the same case under several interleavings.
Step 1: a dispatcher that controls interleaving
Wrap the parallel dispatch point. The wrapper takes a permutation and a barrier policy, records start and finish timestamps per call, and returns results in the original slot order so the agent sees nothing unusual.
# concurrency.py
import asyncio, itertools, time
from dataclasses import dataclass, field
@dataclass
class CallRecord:
name: str
args: dict
started: float
finished: float
result: object
@dataclass
class Schedule:
order: tuple # indices, the order in which calls are allowed to start
stagger: float = 0.0 # seconds between starts; 0.0 = maximally concurrent
log: list = field(default_factory=list)
def scheduled_dispatch(real_dispatch, sched: Schedule):
async def dispatch(calls):
gates = [asyncio.Event() for _ in calls]
async def opener():
for i in sched.order:
gates[i].set()
if sched.stagger:
await asyncio.sleep(sched.stagger)
async def one(i, call):
await gates[i].wait()
t0 = time.monotonic()
res = await real_dispatch(call)
sched.log.append(CallRecord(call.name, call.args, t0, time.monotonic(), res))
return res
opener_task = asyncio.create_task(opener())
results = await asyncio.gather(*(one(i, c) for i, c in enumerate(calls)))
await opener_task
return results
return dispatch
def interleavings(n, limit=6):
"""A few orders, not n!. Listed order, reversed, then a sample."""
perms = [tuple(range(n)), tuple(reversed(range(n)))]
for p in itertools.permutations(range(n)):
if len(perms) >= limit:
break
if p not in perms:
perms.append(p)
return perms
Two knobs matter. order decides who starts first. stagger decides whether calls overlap at all: 0.0 overlaps them fully, which is the case that finds races; a larger value serialises them, which is the case that finds bad ordering assumptions.
Full permutations explode. Six interleavings per case, chosen deterministically and seeded, is enough to catch the common failures and still finish in CI.
Step 2: assert dependency ordering
Some pairs of tools have a real happens-before relationship: you cannot attach_invoice before create_invoice, and you cannot refund_payment before get_payment. If the agent issues both in the same parallel batch, it has asserted an ordering the runtime does not provide. That is an agent bug, not an infrastructure bug, and it is worth failing on.
DEPENDS_ON = {
("attach_invoice", "create_invoice"),
("refund_payment", "get_payment"),
("send_email", "create_ticket"),
}
def batches(log):
"""Group calls that overlapped in time."""
out, current = [], []
for rec in sorted(log, key=lambda r: r.started):
if current and rec.started >= max(c.finished for c in current):
out.append(current); current = []
current.append(rec)
if current:
out.append(current)
return out
def test_no_dependent_pair_in_one_batch(agent, fixtures):
sched = Schedule(order=(0, 1, 2, 3), stagger=0.0)
agent.dispatch = scheduled_dispatch(fixtures.dispatch, sched)
agent.run("Refund the duplicate charge on invoice 8841 and email the customer.")
for group in batches(sched.log):
names = {r.name for r in group}
for after, before in DEPENDS_ON:
assert not (after in names and before in names), \
f"{after} dispatched concurrently with its prerequisite {before}"
This one finds bugs on day one in most agents we look at. The fix is usually not a prompt change; it is declaring the dependency in the tool definitions, or refusing the parallel batch in the runtime when a dependent pair appears.
Step 3: read-after-write staleness
The nastier version is a read and a write on the same entity in one batch. Whether the agent sees the old or new value depends on scheduling, so the final answer is nondeterministic while every tool call is individually correct.
Run the same case under several interleavings and assert the outcome is stable, not that some particular scheduling wins.
import pytest
@pytest.mark.parametrize("order", interleavings(4))
def test_final_state_is_order_invariant(agent, fixtures, order):
sched = Schedule(order=order, stagger=0.0)
agent.dispatch = scheduled_dispatch(fixtures.dispatch, sched)
out = agent.run("Set seat count to 12 and tell me the new monthly total.")
assert fixtures.db.seats("acct_77") == 12
assert out.quoted_total == fixtures.pricing.total(seats=12), \
f"quoted a stale total under order {order}"
Collect the results across the parametrised runs and report them as one number: cases whose end state varied by interleaving. That is the metric to track over time. A pass rate that averages across schedules hides exactly the thing you are measuring.
Step 4: mutual exclusion on writes
Two writes to the same entity in one batch is almost always a defect, even if the end state happens to be fine today.
WRITE_TOOLS = {"create_invoice", "refund_payment", "update_account", "send_email"}
def entity(rec):
for key in ("id", "account_id", "invoice_id", "to"):
if key in rec.args:
return (rec.name, rec.args[key])
return (rec.name, None)
def test_no_concurrent_writes_to_same_entity(agent, fixtures):
sched = Schedule(order=(0, 1, 2, 3), stagger=0.0)
agent.dispatch = scheduled_dispatch(fixtures.dispatch, sched)
agent.run("Apply the credit to account 77 and update the plan.")
for group in batches(sched.log):
seen = set()
for rec in group:
if rec.name not in WRITE_TOOLS:
continue
key = entity(rec)
assert key not in seen, f"two concurrent writes to {key}"
seen.add(key)
Note the overlap with retries: a retried write and a parallel write produce the same duplicate. The difference is that a retry duplicate is caught by an idempotency key, and a concurrent duplicate is not, because both requests are genuinely new.
Step 5: partial-batch failure
Three of four parallel calls succeed. The fourth returns a 409. What the agent does next is the whole question: retry only the failed call, replay all four, or report a half-done state honestly. Force it and assert.
def test_partial_batch_does_not_replay_successes(agent, fixtures):
sched = Schedule(order=(0, 1, 2, 3), stagger=0.0)
dispatch = fail_nth(fixtures.dispatch, index=3, status=409)
agent.dispatch = scheduled_dispatch(dispatch, sched)
agent.run("Create the three line items and post the invoice.")
posted = [r for r in sched.log if r.name == "create_line_item"]
assert len(posted) == 3, "successful calls were replayed after a partial failure"
Replaying the whole batch after one failure is the most common finding here, and the most expensive one, because it duplicates real records.
Step 6: run it as its own suite
Keep concurrency cases separate with their own pass rate, same as fault cases.
pytest tests/evals -m concurrency -q -p no:randomly
Two gates with teeth: zero concurrent writes to the same entity, and zero order-variant outcomes. Both are absolute counts, not thresholds. A percentage target on a suite this small is noise — see measuring noise in agent pass rates for why. Seed the interleaving sampler and pin it, or the suite becomes the flake it was meant to catch.
What this does not tell you
It does not tell you your agent is concurrency-safe. Six interleavings out of n! is a sample, the harness runs on fixtures rather than your real datastore, and production adds contention from other clients that no single-agent test reproduces.
What it does give you is a list of specific ordering assumptions your agent makes and a reproducible case for each. Most of those findings get fixed underneath the agent — declared dependencies, a serialised write path, an idempotency key on a tool that lacked one — which is the usual shape of what we hand back from an agent readiness assessment. The agent is rarely the only thing that needs to change.