A deep-research agent runs for ten minutes, makes forty search and fetch calls, and hands back eight paragraphs with nineteen footnotes. It reads like a good analyst wrote it. That is the problem. Fluent prose is the one thing these agents are reliably good at, and it is the thing a reviewer's eye scores first.
The failure modes we actually find in research-agent traces are not stylistic:
- Unsupported claim. The sentence is plausible, the cited page exists, the page does not say that.
- Citation drift. The claim came from source A, the footnote points at source B, usually the last thing the agent fetched.
- Dead or fabricated URL. The link 404s, or was never fetched during the run at all.
- Stale source. The page says 2023 and the claim is presented as current.
- Coverage gaps. Half the load-bearing sentences carry no citation, and only the safe ones do.
All five are checkable. This tutorial builds a harness that takes a run's report plus its trace, produces claim-citation pairs, and scores each one. You end with three numbers per run — dead-link rate, support rate, citation coverage — that you can trend and gate on.
Step 0: make the run inspectable
You need two things out of the agent: the final report and the list of documents it actually fetched, keyed by URL, with the retrieved text stored. If your fetch tool currently returns text into the context and throws it away, fix that first — write each fetch to a run-scoped store.
# fetch tool wrapper
def fetch(url: str, run_store: dict) -> str:
text = http_get_and_extract(url)
run_store[url] = {"text": text, "fetched_at": now_iso()}
return text
This store is what separates a real citation eval from a link checker. It lets you ask whether the cited page supported the claim at the time of the run, which is the only fair question to ask of the agent. It also makes the eval replayable later, the same way recorded tool responses keep the rest of your agent evals from flaking.
Step 1: extract claim-citation pairs
Require the agent to emit citations in a form you can parse. Inline markers plus a reference list is enough:
Seat licences fell 12% year over year [3].
[3] https://example.com/2026-q2-report
Then split the report into sentences and attach the markers that appear in each one.
import re
CITE = re.compile(r"\[(\d+)\]")
def claim_pairs(report: str, refs: dict[str, str]):
body, _, _ = report.partition("\n[1] ")
for sentence in split_sentences(body):
markers = CITE.findall(sentence)
claim = CITE.sub("", sentence).strip()
if not markers:
yield {"claim": claim, "url": None}
for m in markers:
yield {"claim": claim, "url": refs.get(m)}
If your agent does not emit structured citations, stop here and add them. An unparseable bibliography means every check below turns into a judge's opinion, and you want as few of those as you can get away with.
Step 2: the deterministic checks
Run these before any model touches the data. They are cheap, they are exact, and in our experience they catch the embarrassing failures on their own.
def mechanical_checks(pairs, run_store):
out = []
for p in pairs:
url = p["url"]
out.append({
**p,
"has_citation": url is not None,
"in_run_store": url in run_store, # agent actually fetched it
"resolves": url is not None and head_ok(url), # still live today
})
return out
Three flags, three distinct bugs:
| Flag false | What it means | Where the bug lives |
|---|---|---|
has_citation | load-bearing sentence with no source | prompt and report contract |
in_run_store | cited a URL it never opened | the model invented or recalled it |
resolves | link is dead for your reader | source selection, or time passing |
in_run_store is the one people skip and the one that finds fabrication. An agent that never fetched the page cannot have read it.
Step 3: score support with a judge, narrowly
Only now bring in a model, and give it the smallest possible job: does this passage support this claim? No style, no usefulness, no overall grade.
SUPPORT_PROMPT = """You are checking one claim against one source passage.
CLAIM: {claim}
SOURCE PASSAGE:
{passage}
Answer with one token:
SUPPORTED - the passage states or directly entails the claim
PARTIAL - related, but the claim adds specifics the passage lacks
CONTRADICTED- the passage says something incompatible
ABSENT - the passage does not address the claim
"""
def score_support(pair, run_store, judge):
doc = run_store.get(pair["url"])
if not doc:
return "ABSENT"
passage = top_passages(doc["text"], pair["claim"], k=3)
return judge(SUPPORT_PROMPT.format(claim=pair["claim"], passage=passage))
Two details do most of the work. First, retrieve passages from the stored document rather than pasting the whole page: a long page invites the judge to find something vaguely on topic and call it support. Second, keep PARTIAL as a separate label. Numeric and dated claims fail here constantly — the source says "about a tenth", the report says "12%" — and collapsing that into SUPPORTED hides your most common real error.
This judge is a measurement instrument, so measure it. Label 100 pairs by hand, compare, and report agreement before you trust a single percentage it produces. The procedure is in an unmeasured LLM-as-judge is just a guess, and the drift checks that keep it honest are in your judge drifted and your pass rate lied. If you cannot get agreement above chance on PARTIAL, merge it into CONTRADICTED and say so in the metric definition.
Step 4: three numbers per run
Roll the pairs up:
def run_metrics(scored):
cited = [p for p in scored if p["has_citation"]]
supported = [p for p in cited if p["support"] == "SUPPORTED"]
return {
"citation_coverage": len(cited) / max(len(scored), 1),
"support_rate": len(supported) / max(len(cited), 1),
"dead_or_unfetched_rate": sum(
1 for p in cited if not p["in_run_store"] or not p["resolves"]
) / max(len(cited), 1),
}
Report all three. They trade against each other: an agent that cites nothing scores a perfect support rate, and an agent that staples a footnote to every sentence scores perfect coverage with garbage support. A prompt change that lifts one and drops another is not an improvement, it is a different failure profile.
Fix a small dataset of research questions — twenty is a start, with known-answer questions mixed in where you can verify the number yourself — and run the harness on every prompt, model or tool change. On known-answer cases, add an exact assertion on the final figure. Those are the cases that tell you whether the whole pipeline works, not just whether the footnotes are tidy.
Step 5: gate, carefully
These metrics are noisy. A research agent's search results change between runs, so a 4-point move in support rate on twenty questions may be nothing. Measure the variance before you pick a threshold: run the unchanged agent three to five times and look at the spread, the same way you would for pass-rate noise on any agent eval. Gate on what is clearly outside that band, plus one hard rule that needs no statistics: no merge if any cited URL was never fetched during the run. Fabricated sources are not a threshold question.
What this does not tell you
It does not tell you the report is right. Every claim can be supported by its citation and the whole thing still be wrong, because the agent searched badly and found five sources that agree with each other and not with reality. Source quality, recency and independence are judgment calls, and the only honest way we know to cover them is periodic human review of sampled runs — which is worth running as a standing annotation queue rather than a one-off.
What it does tell you is whether the thing your reader will click matches the thing your agent said. That is the failure that costs you credibility the first time a customer checks a footnote, and unlike report quality, it is a number you can have by Friday. If you want the wider harness around it — dataset curation, calibrated graders, CI gating on your own tooling — that is what our eval suite engineering work builds.