Where the human labels come from: running an annotation queue for agent traces

Every model-graded eval rests on a pile of human labels. The judge prompt gets tuned against them, agreement gets measured against them, and the pass rate you report to leadership inherits whatever quality they had. Most teams we work with have a calibrated-judge plan and no labels, or labels that one engineer produced in a hurry on a Friday and nobody can reproduce.

This tutorial covers the unglamorous half: running an annotation queue over agent traces so that human labels arrive in a known format, at a known rate, with disagreement measured instead of averaged away. The examples use Langfuse annotation queues, but the workflow transfers to Braintrust's review UI, an internal Streamlit page, or a spreadsheet plus a script.

Step 1: write the label schema before you look at traces

A label is only useful if two people applying the rule to the same trace produce the same value. That means the schema comes first, and it should be boring.

Rules that hold up in practice:

• One question per label. resolved_correctly and tone_ok are two labels, not one "quality" score.
• Categorical or boolean, not 1-5. Nobody agrees on the difference between a 3 and a 4. If you need gradations, use named categories with written definitions.
• Add an unclear option. Without it, annotators guess, and guesses look like disagreement about the agent when they are disagreement about the case.
• Define the label on the trajectory, not the reply. For agents the interesting question is usually "did it take the right actions", which means the annotator needs the tool calls in front of them.

A schema for a support agent:

Label	Type	Rule
resolved	boolean	The user's request is fully handled, no follow-up needed from them.
actions_authorized	boolean	Every write tool call was something the user asked for, in the user's own scope.
failure_mode	categorical	none, wrong_tool, missing_lookup, hallucinated_policy, gave_up_early, unclear

Write the rule text down in the repo next to the eval code. The schema is a versioned artifact; when it changes, old labels are on the old version and you do not silently mix them.

Step 2: get the right traces into the queue

Random sampling wastes annotator time on the 80% of traffic that is trivially fine. Sample deliberately, in three buckets:

1. Stratified random across input type, so the dataset still looks like production.
2. Cheap-signal failures: traces where a deterministic check already failed (over tool budget, write without lookup, error span present). These are dense in real failure modes.
3. Judge-disagreement candidates: traces where your current model judge scored near its threshold. These are exactly the cases where calibration is decided.

In Langfuse, you can send traces into a queue from the SDK, which means bucket selection is code you can re-run rather than clicking:

from langfuse import Langfuse

langfuse = Langfuse()

def enqueue(trace_ids, queue_id, batch_note):
    for trace_id in trace_ids:
        langfuse.api.annotation_queues.create_queue_item(
            queue_id=queue_id,
            request={"objectId": trace_id, "objectType": "TRACE"},
        )
    print(f"queued {len(trace_ids)} traces ({batch_note})")

Check the annotation queue docs for the current method and field names; the shape of the call has moved between SDK versions. If your platform has no API for this, an export to CSV plus a local review page is fine. What matters is that the selection query lives in version control with a comment explaining why those traces.

Keep batches small. Fifty traces is a session; five hundred is a month of avoidance. Aim for a standing weekly batch rather than one heroic labeling push.

Step 3: double-label a slice and measure agreement

Here is the step teams skip. Send 20% of every batch to two annotators independently, and compute agreement before you use any of the labels to tune a judge.

Raw agreement is misleading when one class dominates: if 95% of traces are resolved = true, two annotators who always say true agree 95% of the time and have demonstrated nothing. Cohen's kappa corrects for chance agreement:

def cohens_kappa(a, b):
    """a, b: lists of labels from two annotators over the same items."""
    n = len(a)
    observed = sum(1 for x, y in zip(a, b) if x == y) / n
    labels = set(a) | set(b)
    expected = sum(
        (a.count(l) / n) * (b.count(l) / n) for l in labels
    )
    return (observed - expected) / (1 - expected)

We treat kappa below about 0.6 on a label as a schema problem, not an annotator problem. The fix is to pull the disagreeing traces into a fifteen-minute review, find the ambiguity in the rule text, rewrite the rule, and relabel. Two or three rounds of that usually moves a vague label into workable territory — or tells you the label is unanswerable from the trace, which is also a finding.

Record the agreement number per label per batch. It is the honest ceiling on your judge: a model judge cannot meaningfully agree with humans more than humans agree with each other, and a reported judge accuracy above that ceiling is a sign you are measuring against one person's idiosyncrasies.

Step 4: adjudicate, then freeze

Disagreements need a decision, not an average. Pick one reviewer who owns the schema, have them resolve conflicts, and store the resolved label plus the fact that it was contested:

{
  "trace_id": "0f3c…",
  "schema_version": "support-v3",
  "labels": {"resolved": false, "actions_authorized": true,
             "failure_mode": "missing_lookup"},
  "annotators": ["dana", "sam"],
  "contested": true,
  "adjudicated_by": "dana",
  "labeled_at": "2026-09-03"
}

Then commit it. Labels belong in the client's repo as data files, versioned alongside the eval code that consumes them — same reason we ship replayable tool fixtures instead of screenshots. When a judge prompt changes, you re-run against a fixed label set and the comparison means something.

Two practical habits:

• Hold out a slice. Keep 25-30% of labels out of judge-prompt iteration. Tuning a judge against all your labels and then reporting agreement on those same labels is fitting to the test set.
• Re-label a small control set quarterly. Annotator standards drift too, not just judges. If the same 25 traces get different labels in Q1 and Q3, your trend line has a human-shaped bend in it.

What this does not fix

An annotation queue produces labels. It does not produce agreement on what "good" means for your product — that is a conversation with whoever owns the agent, and it is usually the hardest hour of the engagement. It also does not make a judge trustworthy on its own; the labels are the input to calibrating the judge and to watching it for drift afterward.

What it does give you is a defensible answer to "how do you know?" — a written schema, a sampling query, measured inter-annotator agreement and a versioned label set. That is the foundation under a readiness assessment, and it is the part that survives the model you are using today.