Most agent failures are visible in the tool calls: wrong tool, wrong arguments, wrong order, one call too many. Those are structured data, which means you can regression-test them with exact assertions instead of a model's opinion. This tutorial wires Promptfoo up to an agent and turns three common failure classes into checks that run on every change.
Promptfoo is an open-source eval runner: you give it a config listing providers, test cases and assertions, and it runs the matrix and reports pass rates. It is usually pointed at a prompt-plus-model pair, but nothing stops you from pointing it at a whole agent.
Step 1: point Promptfoo at your agent
Initialize a config
npx promptfoo@latest init
This creates promptfooconfig.yaml. The three keys that matter are prompts, providers and tests; the configuration guide documents the rest.
Write a custom provider
An agent is not a prompt template, so the provider is a small script that calls your agent's entry point and returns what it did. The important decision is what to return: return the trajectory, not just the final answer, because the trajectory is where agents fail.
// agent-provider.js
class AgentProvider {
id() {
return 'my-agent';
}
async callApi(prompt) {
const result = await runAgent(prompt); // your agent's entry point
// Return the answer plus the tool-call log your tracer already records.
return {
output: JSON.stringify({
answer: result.answer,
tool_calls: result.toolCalls, // [{ name, arguments }, ...]
}),
};
}
}
module.exports = AgentProvider;
Then reference it from the config:
# promptfooconfig.yaml
prompts:
- '{{input}}'
providers:
- file://agent-provider.js
Run the agent against a staging tool environment or recorded tool responses, never production. If a case can trigger a real send, delete or payment, fix that before writing any assertions.
Step 2: assert on the trajectory
Deterministic checks first
Because the provider returns JSON, a javascript assertion can check tool selection, ordering and call counts exactly:
tests:
- vars:
input: 'Refund order 4832 for the damaged item'
assert:
- type: javascript
value: |
const run = JSON.parse(output);
const calls = run.tool_calls.map((c) => c.name);
return (
calls.includes('lookup_order') &&
calls.indexOf('lookup_order') < calls.indexOf('issue_refund') &&
run.tool_calls.filter((c) => c.name === 'issue_refund').length === 1
);
That one case encodes three expectations: the agent looked the order up, it did so before refunding, and it refunded exactly once. Each maps to a failure class we see in real traces:
| Failure class | Check |
|---|---|
| Wrong tool selected | membership test on the call list |
| Wrong order | compare indexes of the two calls |
| Duplicate irreversible action | count calls to the write tool |
| Runaway loop | cap total tool_calls.length |
Promptfoo ships many more assertion types - equals, contains, regex, is-json and friends are listed in the assertions reference - but for trajectories, a few lines of JavaScript against the parsed output does most of the work.
Share the invariants with defaultTest
Checks that should hold on every case - a tool budget, valid JSON out of the provider - belong in defaultTest so each test inherits them:
defaultTest:
assert:
- type: javascript
value: JSON.parse(output).tool_calls.length <= 8
Save model-graded checks for the fuzzy remainder
Whether the final reply reads well is not a structural question, so use a rubric assertion for it:
- type: llm-rubric
value: The reply acknowledges the damaged item and does not promise a delivery date.
Treat rubric scores with suspicion until the judge has been measured against human labels; an unmeasured judge is a guess. The calibration procedure is its own tutorial.
Step 3: run it, then gate on it
npx promptfoo eval
npx promptfoo view
eval runs the matrix and prints pass rates; view opens a local UI for reading individual failures. In CI, run promptfoo eval on every prompt, model or tool-schema change and gate the merge on its exit status. Start with the rule that no previously passing case may fail; tune thresholds once you have a few weeks of history.
Where this stops being enough
Ten hand-written cases catch regressions on the journeys you thought of. The next steps are the ones that take real effort: mining cases from production traces so the dataset represents actual traffic, building fixtures so external tools replay deterministically, and calibrating every model-graded judge. That is the shape of our eval suite engineering work, and the reasoning behind the layers is in tracing is not the same as knowing your agent works. But the version above is real protection you can have by the end of the day.