Hands-on notes on evals, red-teaming and shipping
September 18, 2026 · Judgeworthy
Your agent's behavior depends on tool names, descriptions and JSON schemas you often do not own. Here is how to snapshot the tool manifest, diff it in CI, and route each kind of change to the right eval.
Read moreSeptember 17, 2026 · Judgeworthy
After a month of prompt tuning against the same 80 cases, your pass rate measures how well you memorized the set. Here is how to split an agent eval dataset into dev and holdout, run them separately, and read the gap.
Read moreSeptember 16, 2026 · Judgeworthy
Every calibrated judge rests on human labels. Here is the workflow that produces them: a written label schema, a deliberate sampling query, double-labeled batches, and inter-annotator agreement measured before anyone tunes a judge.
Read moreSeptember 15, 2026 · Judgeworthy
An agent that guesses on an under-specified request looks great in a demo and files the wrong refund in production. Here is how to build an eval set for abstention and escalation, with two metrics that keep it honest.
Read moreSeptember 14, 2026 · Judgeworthy
Browser and desktop agents take dozens of low-level steps and almost never take the same path twice. Here is how to build an eval that checks the environment afterwards, with a disposable sandbox, checkpoint scoring and a flake budget.
Read more