An eval suite that runs when someone remembers to run it is documentation, not a control. The teams whose agents stop regressing are the ones where a prompt edit cannot reach main without the suite running first. In the LangChain State of Agent Engineering survey, 89% of teams reported having observability and 52% reported having evals; the gap between having evals and gating on them is wider still.
This tutorial wires an agent eval suite into pull requests with Braintrust and GitHub Actions. The parts that matter are not the YAML. They are: what you score, what you compare against, and what actually fails the build.
We covered the same job on Promptfoo in regression-testing an agent's tool calls. Use whichever runner your team already has. The gating logic below is the same either way.
Step 0: prerequisites
A repo containing the agent, a BRAINTRUST_API_KEY stored as a GitHub Actions secret, and pip install braintrust autoevals. You also need somewhere safe to run the agent: a staging tool environment or recorded tool responses. If a test case can trigger a real refund, email or delete, fix that before you wire anything to CI. A suite that mutates production once per pull request is worse than no suite.
Step 1: write the eval as code, not as a config
A Braintrust eval is three things: data, a task, and scorers. Put it in a file the runner can discover, for example evals/refunds.eval.py.
from braintrust import Eval
from evals.cases import REFUND_CASES
from myagent import run_agent
from evals.scorers import lookup_before_refund, tool_budget, no_duplicate_writes
def task(input):
result = run_agent(input, tools=staging_tools())
# Return the trajectory, not just the answer. Agents fail in the middle.
return {
"answer": result.answer,
"tool_calls": [
{"name": c.name, "arguments": c.arguments} for c in result.tool_calls
],
}
Eval(
"support-agent",
data=lambda: REFUND_CASES,
task=task,
scores=[lookup_before_refund, tool_budget, no_duplicate_writes],
)
REFUND_CASES is a list of {"input": ..., "expected": ...} dicts. Keep the cases in version control next to the agent so a pull request that changes behaviour can change the expectations in the same diff, and a reviewer sees both.
Step 2: deterministic scorers first
A scorer in Braintrust returns a name and a score between 0 and 1. Most useful agent scorers are ordinary Python over the tool-call list:
def lookup_before_refund(output, **kwargs):
names = [c["name"] for c in output["tool_calls"]]
if "issue_refund" not in names:
return {"name": "lookup_before_refund", "score": 1}
refund_at = names.index("issue_refund")
return {
"name": "lookup_before_refund",
"score": 1 if "lookup_order" in names[:refund_at] else 0,
}
def tool_budget(output, **kwargs):
calls = len(output["tool_calls"])
return {"name": "tool_budget", "score": 1 if calls <= 8 else 0, "metadata": {"calls": calls}}
def no_duplicate_writes(output, **kwargs):
writes = [c["name"] for c in output["tool_calls"] if c["name"] in WRITE_TOOLS]
return {"name": "no_duplicate_writes", "score": 1 if len(writes) == len(set(writes)) else 0}
Those three map to failure classes we see in real traces: acting before reading, runaway loops, and repeating an irreversible action. They are cheap, they never flake on model temperature, and a failure points at one line of the trajectory. Model-graded scorers from autoevals are useful for the fuzzy remainder - tone, whether the reply answered the question - but do not gate a merge on a judge you have not measured against human labels. The procedure is in an unmeasured LLM-as-judge is just a guess.
A reasonable split for a first gating suite: deterministic scorers block the merge, judge scorers report only.
Step 3: run it locally
braintrust eval evals/
The runner executes the eval files it finds, uploads the results, and prints a link to the experiment. Read a few individual cases before you trust the aggregate. A suite where every case passes on the first run usually means the cases are too easy, not that the agent is good.
Step 4: gate the pull request
# .github/workflows/agent-evals.yml
name: agent-evals
on:
pull_request:
paths:
- 'myagent/**'
- 'prompts/**'
- 'evals/**'
jobs:
evals:
runs-on: ubuntu-latest
timeout-minutes: 20
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: '3.12'
- run: pip install -r requirements.txt braintrust autoevals
- name: Run evals
env:
BRAINTRUST_API_KEY: ${{ secrets.BRAINTRUST_API_KEY }}
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY_EVALS }}
AGENT_ENV: staging
run: braintrust eval evals/
Trigger on the paths that change agent behaviour: agent code, prompts, tool schemas, model configuration, and the eval files themselves. A model version pinned in config counts. So does an MCP server version.
Two things worth doing on day one. Give the job a hard timeout, because a looping agent will otherwise burn twenty minutes on a single case. And use a separate model API key for CI with its own spend limit, so a bad merge costs money in a place you can see.
Step 5: compare against the base commit, not a fixed number
This is the part teams get wrong. "Fail if the average score is below 0.85" produces one of two outcomes: a threshold so low it never fires, or a red build that everyone learns to merge past.
Gate on movement instead. The rule that survives contact with a real team is: no case that passed on the base commit may fail on this branch. Braintrust scores an experiment against a comparison experiment, so the per-case diff is available; the same rule is implementable by hand if you store the previous run's per-case results as a CI artifact and compare.
That framing has two useful consequences. A pull request that legitimately changes behaviour has to update the expected case in the same diff, which makes the change reviewable. And a pull request that quietly breaks an unrelated journey fails, which is the whole point.
Add a second, softer signal for aggregate drift: post the score delta as a pull request comment and let a human read it. Not everything needs to be a gate.
Step 6: deal with non-determinism before it eats your credibility
Agents are not deterministic, and a suite that fails one time in five gets disabled within a month. Four things to do, cheapest first:
- Freeze the environment. Record tool responses as fixtures and replay them, so only the model varies.
- Set temperature to 0 for gating runs. It reduces variance; it does not remove it.
- Run flaky cases several times and score the majority. Three trials is usually enough to tell a real regression from a coin flip.
- Quarantine, do not delete. Move a genuinely unstable case to a non-gating suite with a note, and fix it later. Deleting it hides the instability, which is itself a finding about the agent.
Track the suite's own flake rate as a number. If more than a percent or two of gating cases are non-deterministic, the problem is the harness, not the agent.
What this does not give you
CI gating protects the journeys already in your dataset. It says nothing about traffic you have never seen, and it is not a security control - adversarial cases come from red-teaming, and the ones that matter get replayed here afterwards so a fix is proven rather than asserted. Nor does the suite maintain itself: datasets have to grow from production traces, judges drift, and fixtures rot as tool schemas change.
That maintenance is most of the long-run work, and it is what eval suite engineering and ongoing reliability engineering cover. But the gate above is the step that changes behaviour: once a regression blocks a merge, the suite starts earning its keep.