Hands-on notes on evals, red-teaming and shipping
November 9, 2026 · Judgeworthy
Graders often score longer agent responses higher regardless of correctness. Here is a padding test that measures the effect in an afternoon, and what to do when it shows up.
Read moreNovember 2, 2026 · Judgeworthy
A pairwise judge that prefers slot A will inflate your win rate. Run every pair in both orders, count disagreements as ties, and gate CI on the flip rate.
Read moreOctober 26, 2026 · Judgeworthy
A code-interpreter tool turns your tool surface into arbitrary model-written code on your infrastructure. Here are the probes that measure what that sandbox can reach, and the agent cases that prove it matters.
Read moreOctober 19, 2026 · Judgeworthy
Catalogs grow, selection degrades quietly, and your old eval set stays green. Here is a confusion matrix over tool choice, an ablation to find colliding descriptions, and a CI gate on the catalog itself.
Read moreOctober 12, 2026 · Judgeworthy
Public agentic benchmarks measure someone else's task. Borrow their two good ideas instead: score by database state diff, and report pass^k over repeated trials on your own cases.
Read more