A support agent answers a policy question wrong. Someone reruns it with a bigger model and it passes, so the ticket closes. Two weeks later the same class of failure is back, because the model was never the problem: the search tool returned the wrong document and the model summarized it faithfully.
This is the most common wasted week we see on agents that retrieve. A single pass/fail on the final answer collapses three independent stages into one bit, and one bit cannot tell you whose bug it is. This tutorial splits that bit into three scores — retrieval, grounding, final correctness — and shows how to read the combination.
The three stages
For any turn where the agent calls a search or lookup tool before answering:
| Stage | Question | Ground truth needed |
|---|---|---|
| Retrieval | Did the required evidence come back at all? | The id(s) of the document that contains the answer |
| Grounding | Is every claim in the answer supported by what came back? | Nothing extra; check the answer against the retrieved text |
| Final correctness | Is the answer right? | The expected answer or key facts |
The stages fail independently, and the combination is the diagnosis:
- Retrieval fail, answer wrong → fix the retriever, the index or the query the agent writes. Do not touch the prompt.
- Retrieval pass, grounding fail → the model invented something with the evidence in front of it. Prompt, model or output-format problem.
- Retrieval pass, grounding pass, answer wrong → the evidence was there and used, and it was still wrong. Usually the case is ambiguous, the ground truth is wrong, or the task needs reasoning over several documents.
- Retrieval fail, answer right → the model knew it from pretraining, or guessed. Counts as a pass on the dashboard and as a hazard in the review: the same case fails the day the policy changes.
That last row is why one bit is not enough. A suite that only scores final answers gives full marks to an agent that is not reading its own sources.
Step 1: label evidence, not just answers
Every case needs one extra field beyond input and expected answer: which document actually contains the answer.
# cases/refund-window.yaml
input: "How long does a customer have to return a damaged item?"
expected_answer_contains: ["30 days"]
required_doc_ids: ["policy-returns-v4"]
Pulling these from production traces is the cheap way to build the set: take real questions, look at what the retriever returned, and have someone who knows the domain mark the document that should have come back. The process for mining and de-identifying those traces is in building an agent eval dataset from production traces.
Two rules keep this from rotting. Pin document ids to a version, so an edit to the policy page shows up as a labeling task rather than a mystery regression. And keep a handful of cases where the correct behavior is to answer "I don't know" — required_doc_ids: [] with an expected refusal. Those are the cases that catch a retriever tuned to always return something.
Step 2: score retrieval from the trajectory
You do not need a separate harness for this. The tool calls and their results are already in the trace; the retrieval score is a set operation over the ids that came back.
def retrieval_scores(tool_calls, required_doc_ids, k=5):
returned = []
for call in tool_calls:
if call["name"] in ("search_docs", "lookup_policy"):
returned += [hit["doc_id"] for hit in call["result"]["hits"][:k]]
required = set(required_doc_ids)
if not required:
return {"recall_at_k": None, "retrieved_any": bool(returned)}
hit_ids = required & set(returned)
ranks = [returned.index(d) + 1 for d in hit_ids]
return {
"recall_at_k": len(hit_ids) / len(required),
"first_hit_rank": min(ranks) if ranks else None,
}
recall_at_k is the one that decides whether the stage passed: if the evidence never arrived, nothing downstream is the model's fault. first_hit_rank is the one that tells you what to do about it. Recall of 1.0 with the required document sitting at rank 5 out of 5 is a reranking problem waiting to become an outage the day someone adds documents to the index.
While you are in the trajectory, record the query the agent actually sent. Agents rewrite user questions before searching, and a surprising share of retrieval failures are the agent searching for something no one asked about. That column pays for itself the first time you sort by it.
Step 3: score grounding against what came back
Grounding is the one stage that needs a model, because it is a claim-by-claim comparison of free text. Keep the judge's job small: give it the answer and the retrieved chunks only — never the ground truth, never the question's expected answer — and ask for a verdict per claim.
GROUNDING_PROMPT = """You are checking whether an answer is supported by evidence.
EVIDENCE (the only source of truth):
{evidence}
ANSWER:
{answer}
Split the ANSWER into factual claims. For each claim, output SUPPORTED,
CONTRADICTED, or NOT_IN_EVIDENCE, with the sentence of evidence you used.
Generic pleasantries are not claims. Return JSON:
{{"claims": [{{"claim": "...", "verdict": "...", "evidence_quote": "..."}}]}}
"""
def grounding_score(judge_output):
claims = judge_output["claims"]
if not claims:
return 1.0
return sum(c["verdict"] == "SUPPORTED" for c in claims) / len(claims)
Requiring a quote is the part that matters. A judge that has to point at the sentence it relied on is much harder to talk into a lazy pass, and the quotes give a human something to audit in thirty seconds.
This is still a model grading a model, so it is a guess until you measure it. Label 50 to 100 answers by hand, compare, and report the agreement number alongside the score. The procedure — including what to do when agreement comes back at 0.6 — is in an unmeasured LLM-as-judge is just a guess. If the judge cannot be calibrated to something you trust, ship the deterministic stages and leave grounding as a review queue rather than a gate.
Step 4: report the combination, not the average
Do not average the three scores. A mean of 0.8 hides which stage is broken, which is the whole point of splitting them. Report the contingency table:
answer right answer wrong
retrieval pass, grounded 142 11
retrieval pass, ungrounded 6 18
retrieval fail 9 37
Read it as a work queue. Forty-six cases here are retrieval bugs; the model is not involved. Twenty-four are grounding bugs. Eleven are genuinely hard cases worth reading one by one. The nine top-right — right answer, missing evidence — are the ones to check first, because they are passing today for a reason that will not hold.
In CI, gate each stage separately with its own threshold: retrieval recall and grounding rate as two independent checks, plus the no-previously-passing-case-may-fail rule. Wiring that into a pull request is covered in gating a pull request on agent evals. If you already write scores to Langfuse, emit all three as separate named scores on the trace so the split survives into the dashboards — see scoring agent traces in Langfuse.
What this does not tell you
Stage attribution assumes the stages are clean. Multi-hop questions, where document A is only findable after reading document B, do not fit a single required_doc_ids list; label the hops separately or keep those cases out of the retrieval score. And none of this says anything about whether the retrieved documents should have been visible to that user in the first place — that is an authorization probe, not a quality metric.
What it does buy you is an end to the argument. When the next failure lands, the table says whether it belongs to the index or the prompt, and you stop swapping models to fix search bugs. Building this out over a real corpus, with the labeling and the calibration done properly, is eval suite engineering work — but the three-column version above is a day, and it is the day that stops the wasted week.