October 19, 2026 · Judgeworthy
Catalogs grow, selection degrades quietly, and your old eval set stays green. Here is a confusion matrix over tool choice, an ablation to find colliding descriptions, and a CI gate on the catalog itself.
Read moreOctober 12, 2026 · Judgeworthy
Public agentic benchmarks measure someone else's task. Borrow their two good ideas instead: score by database state diff, and report pass^k over repeated trials on your own cases.
Read moreOctober 5, 2026 · Judgeworthy
A voice agent is your text agent behind a noisy channel. Here is how to perturb transcripts the way ASR actually fails, assert on the tool calls, and measure barge-in and latency separately.
Read moreSeptember 21, 2026 · Judgeworthy
Agent SDKs fire tool calls in parallel by default, which turns ordering assumptions into intermittent bugs. Here is a deterministic interleaving harness and four pytest evals that make those failures reproducible.
Read moreSeptember 21, 2026 · Judgeworthy
A research agent's report reads well whether or not it is true. Here is a harness that pulls every claim-citation pair, refetches the source, and scores support, so you get three numbers instead of an impression.
Read moreSeptember 21, 2026 · Judgeworthy
Every dataset tutorial assumes you already have traces. Before launch you have none. Here is how we generate, deduplicate and human-review a starter eval set from the agent's tool schemas and a failure taxonomy.
Read moreSeptember 18, 2026 · Judgeworthy
Your agent's behavior depends on tool names, descriptions and JSON schemas you often do not own. Here is how to snapshot the tool manifest, diff it in CI, and route each kind of change to the right eval.
Read moreSeptember 17, 2026 · Judgeworthy
After a month of prompt tuning against the same 80 cases, your pass rate measures how well you memorized the set. Here is how to split an agent eval dataset into dev and holdout, run them separately, and read the gap.
Read moreSeptember 16, 2026 · Judgeworthy
Every calibrated judge rests on human labels. Here is the workflow that produces them: a written label schema, a deliberate sampling query, double-labeled batches, and inter-annotator agreement measured before anyone tunes a judge.
Read moreSeptember 15, 2026 · Judgeworthy
An agent that guesses on an under-specified request looks great in a demo and files the wrong refund in production. Here is how to build an eval set for abstention and escalation, with two metrics that keep it honest.
Read moreSeptember 14, 2026 · Judgeworthy
Browser and desktop agents take dozens of low-level steps and almost never take the same path twice. Here is how to build an eval that checks the environment afterwards, with a disposable sandbox, checkpoint scoring and a flake budget.
Read moreSeptember 14, 2026 · Judgeworthy
Your eval suite runs against tools that always work. Production tools time out, rate-limit and return half a record. Here is a fault-injecting wrapper, six failure modes worth testing, and pytest assertions on what the agent did next.
Read moreSeptember 14, 2026 · Judgeworthy
Long-running agents compact their history to stay inside the context window. Compaction is lossy, and what it drops is usually a constraint. Here is a pytest harness that scores a compactor on fact retention and then checks the agent still obeys.
Read moreSeptember 11, 2026 · Judgeworthy
You calibrated the judge once. Then the provider rotated the model, someone edited the rubric, and your pass rate moved without the agent changing. Here is a frozen gold set, a nightly agreement check and the alert that fires before you trust a bad number.
Read moreSeptember 9, 2026 · Judgeworthy
Your offline suite passed and you still do not know how the new version behaves on real traffic. Here is a shadow runner that replays live requests against the candidate, scores both versions online, and a canary rule that says when to cut over.
Read moreSeptember 9, 2026 · Judgeworthy
An eval suite that calls live tools measures the internet as much as the agent. Here is a cassette-based record/replay harness in pytest, plus the nightly contract check that keeps the recordings honest.
Read moreSeptember 9, 2026 · Judgeworthy
A failing case on a RAG-backed agent tells you nothing about which stage broke. Here is a per-stage eval that scores retrieval, grounding and the final answer separately, so the failure lands on one team's desk.
Read moreSeptember 9, 2026 · Judgeworthy
A model deprecation notice is a regression test you did not write. Here is a paired migration harness in pytest: replay production cases against both models, diff trajectories case by case, and report what actually changed.
Read moreSeptember 4, 2026 · Judgeworthy
Hand-written test cases test what you imagined. Here is how to sample, dedupe, label and split a 60-case eval dataset from the traces your agent already produced - and what a pass rate on 60 cases can honestly tell you.
Read moreSeptember 4, 2026 · Judgeworthy
An eval suite nobody runs is documentation. Here is how to run trajectory evals on every pull request with Braintrust, compare against the base commit, and fail the build on a regression instead of an absolute score.
Read moreSeptember 4, 2026 · Judgeworthy
Agent evals are nondeterministic, so a pass rate that moves from 82% to 87% may be nothing. Here is how to measure your suite's flake rate, size a run, and use a paired test before you call a change an improvement.
Read moreSeptember 4, 2026 · Judgeworthy
Correctness evals say nothing about an agent that takes forty steps to do a three-step job. Here is how to turn step count, tool calls, tokens and latency into hard assertions, and how to test the loop on purpose.
Read moreAugust 25, 2026 · Judgeworthy
Traces tell you what your agent did. Scores turn that into a quality signal you can chart, threshold and argue from. Here is how to add them with the Python SDK.
Read moreAugust 13, 2026 · Judgeworthy
Most agent teams can see every step their agent takes and still cannot say whether it took the right ones.
Read moreJuly 12, 2026 · Judgeworthy
Your security team tests the application; nobody has tested what happens when a support ticket tells the agent to email the customer list.
Read more