Hands-on notes on evals, red-teaming and shipping
September 9, 2026 · Judgeworthy
An eval suite that calls live tools measures the internet as much as the agent. Here is a cassette-based record/replay harness in pytest, plus the nightly contract check that keeps the recordings honest.
Read moreSeptember 9, 2026 · Judgeworthy
A failing case on a RAG-backed agent tells you nothing about which stage broke. Here is a per-stage eval that scores retrieval, grounding and the final answer separately, so the failure lands on one team's desk.
Read moreSeptember 9, 2026 · Judgeworthy
A model deprecation notice is a regression test you did not write. Here is a paired migration harness in pytest: replay production cases against both models, diff trajectories case by case, and report what actually changed.
Read moreSeptember 9, 2026 · Judgeworthy
Give an agent persistent memory and a prompt injection stops being a single-run problem. Here is a pytest harness that writes a payload in session one and checks what the agent does in session two.
Read moreSeptember 4, 2026 · Judgeworthy
Evals that read your framework's objects break when you change frameworks. Here is how to emit GenAI semantic-convention spans for model and tool calls, export them once, and write trajectory assertions that outlive your stack.
Read more