Hands-on notes on evals, red-teaming and shipping
September 14, 2026 · Judgeworthy
Your eval suite runs against tools that always work. Production tools time out, rate-limit and return half a record. Here is a fault-injecting wrapper, six failure modes worth testing, and pytest assertions on what the agent did next.
Read moreSeptember 14, 2026 · Judgeworthy
Long-running agents compact their history to stay inside the context window. Compaction is lossy, and what it drops is usually a constraint. Here is a pytest harness that scores a compactor on fact retention and then checks the agent still obeys.
Read moreSeptember 11, 2026 · Judgeworthy
You calibrated the judge once. Then the provider rotated the model, someone edited the rubric, and your pass rate moved without the agent changing. Here is a frozen gold set, a nightly agreement check and the alert that fires before you trust a bad number.
Read moreSeptember 10, 2026 · Judgeworthy
A refusal layer that stops 100% of injections and 8% of real work is a bad trade you cannot see with attack cases alone. Here is how to score a guardrail on both halves and pick a threshold from data.
Read moreSeptember 9, 2026 · Judgeworthy
Your offline suite passed and you still do not know how the new version behaves on real traffic. Here is a shadow runner that replays live requests against the candidate, scores both versions online, and a canary rule that says when to cut over.
Read more