Hands-on notes on evals, red-teaming and shipping
September 4, 2026 · Judgeworthy
Single-turn cases miss the failures that only appear on turn four: dropped constraints, stale state, repeated write calls. Here is a simulated-user harness in pytest that runs whole conversations and asserts on them.
Read moreSeptember 4, 2026 · Judgeworthy
When you split one agent into a router and specialists, you add three new failure modes: wrong routing, context dropped at the handoff, and agents bouncing work back and forth. Here is a pytest harness that scores all three.
Read moreSeptember 4, 2026 · Judgeworthy
Hand-written test cases test what you imagined. Here is how to sample, dedupe, label and split a 60-case eval dataset from the traces your agent already produced - and what a pass rate on 60 cases can honestly tell you.
Read moreSeptember 4, 2026 · Judgeworthy
An eval suite nobody runs is documentation. Here is how to run trajectory evals on every pull request with Braintrust, compare against the base commit, and fail the build on a regression instead of an absolute score.
Read moreSeptember 4, 2026 · Judgeworthy
Agent evals are nondeterministic, so a pass rate that moves from 82% to 87% may be nothing. Here is how to measure your suite's flake rate, size a run, and use a paired test before you call a change an improvement.
Read more