Every eval harness starts with the same plumbing problem: the eval needs to know what the agent did, and that record lives in whatever shape your framework happened to emit. Swap LangChain for a hand-rolled loop, or move from Langfuse to Braintrust, and the extraction code you wrote to read trajectories breaks. The fix is to stop reading your framework's private objects and start reading OpenTelemetry spans with GenAI semantic conventions on them.
This tutorial instruments an agent so that each model call and each tool call becomes a span with predictable attribute names, exports those spans to a backend that speaks OTLP, and then writes trajectory assertions that read the spans instead of the framework. The payoff is that the assertions survive a framework migration and a backend migration.
One caveat up front: the GenAI conventions are still moving. Attribute names for agent and tool spans have changed between releases, and they will change again. Pin the semconv version you code against, keep the mapping in one module, and expect to edit that module once or twice a year. That is still cheaper than re-writing every eval when you change frameworks.
Step 1: emit spans with GenAI attributes
The two span kinds evals care about
For agent evals, two operations carry almost all the signal:
- Inference spans - one per model call.
gen_ai.operation.nameischat, plusgen_ai.request.model,gen_ai.usage.input_tokens,gen_ai.usage.output_tokens. - Tool execution spans - one per tool call.
gen_ai.operation.nameisexecute_tool, plusgen_ai.tool.nameand, if you choose to record it, the arguments.
A run is a parent span with those two kinds of children in the order they happened. That ordered list is the trajectory. Everything in this tutorial follows from that.
Instrumenting a tool call
If you use an off-the-shelf instrumentation package (OpenInference, OpenLLMetry, or the vendor SDK's own OTel exporter), inference spans usually come for free. Tool spans are the ones people forget, and they are the ones trajectory evals need most, so wrap the tool dispatcher yourself:
from opentelemetry import trace
tracer = trace.get_tracer("agent")
def call_tool(name: str, arguments: dict):
with tracer.start_as_current_span(
f"execute_tool {name}",
attributes={
"gen_ai.operation.name": "execute_tool",
"gen_ai.tool.name": name,
# Arguments are useful and sometimes sensitive. Redact before setting.
"gen_ai.tool.call.arguments": json.dumps(redact(arguments)),
},
) as span:
result = TOOLS[name](**arguments)
span.set_attribute("gen_ai.tool.call.result.length", len(str(result)))
return result
Two decisions worth making deliberately:
- Redact at the span boundary, not at the exporter. Whatever you put in an attribute is going to a third-party backend. Write one
redact()and use it everywhere. - Record lengths and identifiers, not whole payloads, unless you actually need the payload for grading. A 40 KB tool result inside a span attribute is expensive and rarely read.
Tag the run so you can find it later
Set a session or thread identifier and your own case identifier on the parent span. Without it you will be joining evals to traces by timestamp, which fails the first time you run the suite in parallel.
with tracer.start_as_current_span(
"invoke_agent",
attributes={
"gen_ai.operation.name": "invoke_agent",
"gen_ai.agent.name": "support-agent",
"eval.case_id": case_id, # your own namespace, not gen_ai.*
"eval.suite_run_id": run_id,
},
):
...
Keep your own attributes in your own namespace. Do not invent new gen_ai.* keys; a future semconv release may define them differently.
Step 2: export the same spans twice
OTLP means one instrumentation and more than one consumer. Point the SDK at a collector, and fan out from there: your existing APM, your eval backend, and a local file exporter for tests.
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
provider = TracerProvider()
provider.add_span_processor(
BatchSpanProcessor(OTLPSpanExporter(endpoint=os.environ["OTEL_EXPORTER_OTLP_ENDPOINT"]))
)
trace.set_tracer_provider(provider)
Langfuse accepts OTLP directly, so the same spans that go to your APM can land in the trace view you already score in. If you prefer Braintrust or an OpenTelemetry-native store, the agent code does not change - only the endpoint does. That is the whole point of doing it this way.
For the eval suite itself, use an in-memory exporter so a test can read spans without a network round trip:
from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
exporter = InMemorySpanExporter()
provider.add_span_processor(SimpleSpanProcessor(exporter))
Step 3: turn spans into a trajectory your assertions can read
One adapter function, one shape. Everything downstream depends on this and nothing else depends on the framework.
def trajectory(spans):
"""Ordered list of {kind, name, arguments} from GenAI spans."""
steps = []
for span in sorted(spans, key=lambda s: s.start_time):
op = span.attributes.get("gen_ai.operation.name")
if op == "execute_tool":
steps.append({
"kind": "tool",
"name": span.attributes.get("gen_ai.tool.name"),
"arguments": json.loads(span.attributes.get("gen_ai.tool.call.arguments", "{}")),
})
elif op == "chat":
steps.append({
"kind": "model",
"name": span.attributes.get("gen_ai.request.model"),
"input_tokens": span.attributes.get("gen_ai.usage.input_tokens", 0),
})
return steps
Now the assertions are ordinary Python against ordinary dicts:
def test_refund_requires_lookup_first(agent, exporter):
agent.run("Refund order 4832, it arrived damaged")
steps = trajectory(exporter.get_finished_spans())
tools = [s["name"] for s in steps if s["kind"] == "tool"]
assert "lookup_order" in tools
assert tools.index("lookup_order") < tools.index("issue_refund")
assert tools.count("issue_refund") == 1
def test_token_budget(agent, exporter):
agent.run("What is the status of order 4832?")
steps = trajectory(exporter.get_finished_spans())
assert sum(s.get("input_tokens", 0) for s in steps) < 30_000
assert len([s for s in steps if s["kind"] == "model"]) <= 6
The second test is the one teams skip and then regret. Loop and cost blowups do not show up in output-quality scores; they show up as a step count that crept from 4 to 19 after a prompt change. Once tokens and step counts are in the span record, that regression is a one-line assertion.
| Signal in the spans | Failure it catches |
|---|---|
| Ordered tool names | Wrong tool, wrong order |
| Count of write-tool calls | Duplicate irreversible action |
| Number of model spans | Reasoning loops |
| Summed input tokens | Context bloat, cost regressions |
| Span status / exception events | Silent tool errors the agent papered over |
That last row matters more than it looks. An agent that swallows a failed lookup_order and answers anyway looks fine in the final response and looks obviously broken in the spans.
What this does not give you
Instrumentation is plumbing, not judgment. Spans tell you what happened; they do not tell you whether what happened was correct for the task. You still need cases with expected trajectories, and for the fuzzy parts you still need a grader whose agreement with human labels you have actually measured. Nothing about OpenTelemetry makes an unmeasured judge trustworthy.
Two more limits worth stating plainly:
- Conventions drift. Attributes get renamed and promoted between semconv releases. Isolate the mapping in
trajectory()so a rename is one diff, and pin the semconv version in your dependency file. - Spans are sampled. Head sampling in production means the trace behind an incident may not exist. For evals this is fine - sample everything in the eval run - but do not assume your production trace store has the case you want to reproduce.
The check we run
When we do an Agent Readiness Assessment, one of the first things we look for is whether the tool calls are in the trace at all. Teams with model-call tracing and no tool spans can see the agent thinking and not what it did. Adding execute_tool spans is usually a half-day of work, and it is what makes everything after it - trajectory evals, regression gating, red-team replay - possible at all.
If you have tracing in place and no regression suite, tell us about the agent and its tools and we will reply within one business day with what we would test first.