November 9, 2026 · Judgeworthy
Graders often score longer agent responses higher regardless of correctness. Here is a padding test that measures the effect in an afternoon, and what to do when it shows up.
Read moreNovember 2, 2026 · Judgeworthy
A pairwise judge that prefers slot A will inflate your win rate. Run every pair in both orders, count disagreements as ties, and gate CI on the flip rate.
Read moreOctober 26, 2026 · Judgeworthy
A code-interpreter tool turns your tool surface into arbitrary model-written code on your infrastructure. Here are the probes that measure what that sandbox can reach, and the agent cases that prove it matters.
Read moreOctober 19, 2026 · Judgeworthy
Catalogs grow, selection degrades quietly, and your old eval set stays green. Here is a confusion matrix over tool choice, an ablation to find colliding descriptions, and a CI gate on the catalog itself.
Read moreOctober 12, 2026 · Judgeworthy
Public agentic benchmarks measure someone else's task. Borrow their two good ideas instead: score by database state diff, and report pass^k over repeated trials on your own cases.
Read moreOctober 5, 2026 · Judgeworthy
A voice agent is your text agent behind a noisy channel. Here is how to perturb transcripts the way ASR actually fails, assert on the tool calls, and measure barge-in and latency separately.
Read moreSeptember 28, 2026 · Judgeworthy
Your injection suite watches tool calls. The easiest way out of an agent is a rendered markdown image that the browser fetches before anyone reads the answer. Here is a channel inventory, a canary, and a pytest suite that asserts on destinations rather than payloads.
Read moreSeptember 21, 2026 · Judgeworthy
Agent SDKs fire tool calls in parallel by default, which turns ordering assumptions into intermittent bugs. Here is a deterministic interleaving harness and four pytest evals that make those failures reproducible.
Read moreSeptember 21, 2026 · Judgeworthy
A research agent's report reads well whether or not it is true. Here is a harness that pulls every claim-citation pair, refetches the source, and scores support, so you get three numbers instead of an impression.
Read moreSeptember 21, 2026 · Judgeworthy
Every dataset tutorial assumes you already have traces. Before launch you have none. Here is how we generate, deduplicate and human-review a starter eval set from the agent's tool schemas and a failure taxonomy.
Read moreSeptember 18, 2026 · Judgeworthy
Your agent's behavior depends on tool names, descriptions and JSON schemas you often do not own. Here is how to snapshot the tool manifest, diff it in CI, and route each kind of change to the right eval.
Read moreSeptember 17, 2026 · Judgeworthy
After a month of prompt tuning against the same 80 cases, your pass rate measures how well you memorized the set. Here is how to split an agent eval dataset into dev and holdout, run them separately, and read the gap.
Read moreSeptember 16, 2026 · Judgeworthy
Every calibrated judge rests on human labels. Here is the workflow that produces them: a written label schema, a deliberate sampling query, double-labeled batches, and inter-annotator agreement measured before anyone tunes a judge.
Read moreSeptember 15, 2026 · Judgeworthy
An agent that guesses on an under-specified request looks great in a demo and files the wrong refund in production. Here is how to build an eval set for abstention and escalation, with two metrics that keep it honest.
Read moreSeptember 14, 2026 · Judgeworthy
Browser and desktop agents take dozens of low-level steps and almost never take the same path twice. Here is how to build an eval that checks the environment afterwards, with a disposable sandbox, checkpoint scoring and a flake budget.
Read moreSeptember 14, 2026 · Judgeworthy
Your eval suite runs against tools that always work. Production tools time out, rate-limit and return half a record. Here is a fault-injecting wrapper, six failure modes worth testing, and pytest assertions on what the agent did next.
Read moreSeptember 14, 2026 · Judgeworthy
Long-running agents compact their history to stay inside the context window. Compaction is lossy, and what it drops is usually a constraint. Here is a pytest harness that scores a compactor on fact retention and then checks the agent still obeys.
Read moreSeptember 11, 2026 · Judgeworthy
You calibrated the judge once. Then the provider rotated the model, someone edited the rubric, and your pass rate moved without the agent changing. Here is a frozen gold set, a nightly agreement check and the alert that fires before you trust a bad number.
Read moreSeptember 10, 2026 · Judgeworthy
A refusal layer that stops 100% of injections and 8% of real work is a bad trade you cannot see with attack cases alone. Here is how to score a guardrail on both halves and pick a threshold from data.
Read moreSeptember 9, 2026 · Judgeworthy
Your offline suite passed and you still do not know how the new version behaves on real traffic. Here is a shadow runner that replays live requests against the candidate, scores both versions online, and a canary rule that says when to cut over.
Read moreSeptember 9, 2026 · Judgeworthy
An eval suite that calls live tools measures the internet as much as the agent. Here is a cassette-based record/replay harness in pytest, plus the nightly contract check that keeps the recordings honest.
Read moreSeptember 9, 2026 · Judgeworthy
A failing case on a RAG-backed agent tells you nothing about which stage broke. Here is a per-stage eval that scores retrieval, grounding and the final answer separately, so the failure lands on one team's desk.
Read moreSeptember 9, 2026 · Judgeworthy
A model deprecation notice is a regression test you did not write. Here is a paired migration harness in pytest: replay production cases against both models, diff trajectories case by case, and report what actually changed.
Read moreSeptember 9, 2026 · Judgeworthy
Give an agent persistent memory and a prompt injection stops being a single-run problem. Here is a pytest harness that writes a payload in session one and checks what the agent does in session two.
Read moreSeptember 4, 2026 · Judgeworthy
Evals that read your framework's objects break when you change frameworks. Here is how to emit GenAI semantic-convention spans for model and tool calls, export them once, and write trajectory assertions that outlive your stack.
Read moreSeptember 4, 2026 · Judgeworthy
Single-turn cases miss the failures that only appear on turn four: dropped constraints, stale state, repeated write calls. Here is a simulated-user harness in pytest that runs whole conversations and asserts on them.
Read moreSeptember 4, 2026 · Judgeworthy
When you split one agent into a router and specialists, you add three new failure modes: wrong routing, context dropped at the handoff, and agents bouncing work back and forth. Here is a pytest harness that scores all three.
Read moreSeptember 4, 2026 · Judgeworthy
Hand-written test cases test what you imagined. Here is how to sample, dedupe, label and split a 60-case eval dataset from the traces your agent already produced - and what a pass rate on 60 cases can honestly tell you.
Read moreSeptember 4, 2026 · Judgeworthy
An eval suite nobody runs is documentation. Here is how to run trajectory evals on every pull request with Braintrust, compare against the base commit, and fail the build on a regression instead of an absolute score.
Read moreSeptember 4, 2026 · Judgeworthy
Agent evals are nondeterministic, so a pass rate that moves from 82% to 87% may be nothing. Here is how to measure your suite's flake rate, size a run, and use a paired test before you call a change an improvement.
Read moreSeptember 4, 2026 · Judgeworthy
Correctness evals say nothing about an agent that takes forty steps to do a three-step job. Here is how to turn step count, tool calls, tokens and latency into hard assertions, and how to test the loop on purpose.
Read moreSeptember 4, 2026 · Judgeworthy
Injection is not the only way an agent crosses a permission boundary. Sometimes the user just asks. Here is a pytest harness that probes authorization directly and turns every crossing into a replayable case.
Read moreAugust 27, 2026 · Judgeworthy
Hostile text arriving through an MCP tool is the injection path we find most often. Here is a stub server, a case file and pytest assertions that turn it into a suite you can re-run after every fix.
Read moreAugust 25, 2026 · Judgeworthy
Traces tell you what your agent did. Scores turn that into a quality signal you can chart, threshold and argue from. Here is how to add them with the Python SDK.
Read moreAugust 21, 2026 · Judgeworthy
Most agent failures are visible in the tool calls, which are structured data. Here is how to assert on them exactly with Promptfoo and gate a merge on the result.
Read moreAugust 13, 2026 · Judgeworthy
Most agent teams can see every step their agent takes and still cannot say whether it took the right ones.
Read moreAugust 1, 2026 · Judgeworthy
Model-graded evals are unavoidable for agents, and most of them are wrong in ways nobody has checked.
Read more