Teams keep shipping a text agent, then putting a phone number in front of it. The reasoning loop is the same, the tools are the same, and the eval suite is the same - which is the problem. A voice agent takes a different input: a transcript produced by a speech recognizer that drops words, splits turns in the wrong place, and renders "eight four three two" four different ways depending on the audio.
So the suite has to cover two things your text suite does not: how the agent behaves when the transcript is wrong, and how the pipeline behaves in time. Those are separate problems with separate tests. This tutorial covers both, and neither requires a voice eval product.
Split the system before you test it
A typical stack is ASR -> agent -> TTS, with a turn-taking layer deciding when the user is done talking. Four failure classes, four owners:
| Failure | Where it lives | How you test it |
|---|---|---|
| Wrong order number, wrong name, wrong date | ASR | transcript perturbation cases |
| Confident action on a misheard request | agent | tool-call assertions |
| Agent talks over the caller, or waits three seconds | turn-taking | span timing from live or replayed calls |
| Reply is correct but unspeakable (markdown, a URL, a 40-word sentence) | output formatting | deterministic checks on the text |
Most of the damage sits in row two. The ASR will be wrong sometimes; that is a given. What matters is whether the agent notices.
Step 1: build perturbation cases that match real ASR errors
Do not add random typos. Speech recognizers fail in specific ways, and those are the ones worth encoding. Take twenty real transcripts from production calls, then generate variants:
# perturb.py - ASR-shaped corruptions, not random noise
import re
HOMOPHONES = {"to": "two", "for": "four", "their": "there", "eight": "ate"}
def digits_as_words(t: str) -> str:
words = {"0": "oh", "1": "one", "2": "two", "3": "three", "4": "four"}
return re.sub(r"\d", lambda m: words.get(m.group(), m.group()) + " ", t)
def drop_short_words(t: str) -> str:
return " ".join(w for w in t.split() if len(w) > 2)
def homophone_swap(t: str) -> str:
return " ".join(HOMOPHONES.get(w.lower(), w) for w in t.split())
def truncate_turn(t: str) -> str:
# endpointing fired early: caller was still talking
return " ".join(t.split()[: max(3, len(t.split()) // 2)])
PERTURBATIONS = [digits_as_words, drop_short_words, homophone_swap, truncate_turn]
Four variants per seed transcript gives you eighty cases from twenty calls. Store them as data - seed text, perturbation name, and the expected behaviour - the same way you would store any other eval dataset mined from production traces.
The expected behaviour is the part people get wrong. For digits_as_words the agent should still resolve the order. For truncate_turn the correct answer is usually not an action at all: it should ask. Encode that.
Step 2: assert on tool calls, not on the reply text
The check you care about is whether a garbled transcript produced a confident write. Run the agent on the perturbed transcript and assert against the trajectory:
# test_asr_robustness.py
import pytest
from perturb import PERTURBATIONS
from agent import run_agent # returns .tool_calls and .reply
WRITE_TOOLS = {"issue_refund", "cancel_order", "update_address", "transfer_call"}
@pytest.mark.parametrize("case", load_cases())
@pytest.mark.parametrize("perturb", PERTURBATIONS, ids=lambda f: f.__name__)
def test_no_unconfirmed_write(case, perturb):
run = run_agent(perturb(case["transcript"]), tools=stub_tools())
writes = [c for c in run.tool_calls if c.name in WRITE_TOOLS]
if writes:
# a write is allowed only if the agent read back the key field first
assert case["order_id"] in run.reply_before_first_write, (
f"wrote {writes[0].name} without confirming the order id"
)
That single invariant - no irreversible tool call without reading the identifier back to the caller - catches more voice-specific bugs than any rubric about tone. Add the usual trajectory assertions on top: right tool, right order, no duplicate writes. The mechanics are the same as regression-testing tool calls with Promptfoo; only the inputs changed.
Run the tools as stubs. A perturbation suite that fires eighty refunds against a live sandbox will teach you something you did not want to learn. Record and replay them as described in deterministic tool replay.
Step 3: score abstention, because that is the correct answer
On truncated and heavily corrupted inputs, the pass condition is a clarifying question. Measure both directions or you will optimize into an agent that asks about everything:
- Unsafe-action rate: fraction of corrupted cases ending in a write tool call. Target zero on truncated turns.
- Over-asking rate: fraction of clean transcripts where the agent asks instead of acting. Track it on an unperturbed control set in the same run.
Two numbers, one table, reported per perturbation type so you can see which corruption class the agent cannot handle. We walk through the scoring in escalation evals; the only voice-specific part is that the trigger is channel noise rather than ambiguity.
Step 4: measure timing from spans, separately
Turn-taking is not a model-quality question and does not belong in the same suite. Instrument the pipeline and pull three durations per turn: endpoint-to-first-token, first-token-to-first-audio, and total user-silence. Report p50 and p95, not the mean - the mean hides exactly the calls where someone hung up. If you already emit OpenTelemetry GenAI spans, these are span deltas, and the gate is a threshold in CI on replayed calls.
Barge-in deserves one explicit test: play audio while the agent is speaking and assert that TTS stops and the interrupted text does not stay in context as if it were delivered. Agents that "remember" saying something the caller never heard produce some of the strangest transcripts we see.
What we do not test this way
Transcript perturbation is not an audio test. It will not catch an accent the recognizer handles badly, a codec problem, or a diarization failure on a three-party call. Those need real audio fixtures through the real ASR, and word error rate measured per caller cohort. Do that too - but do it as a separate, smaller suite, because audio fixtures are slow and expensive to maintain, and 80% of the production incidents we are asked about are the agent acting confidently on a transcript it should have questioned.
Also worth saying: a voice channel widens the red-team surface rather than changing it. Injection arrives through the same tool outputs and documents, and a caller reading an instruction aloud is still an instruction. If a voice agent is going live, the red-team battery runs against the text layer, with the perturbation set layered on top.
If you want a second pair of eyes on a voice agent before it answers real calls, that is what an agent readiness assessment is for.