If your agent's runs already land in Langfuse, you are one step from a quality signal: scores. A score attaches a named value to a trace or a single observation - numeric, boolean, categorical or text - and once scores exist you can chart pass rates over time, compare model versions, and answer "did Tuesday's prompt change make refunds worse?" with a number instead of a shrug.
This tutorial adds two custom scores to an agent's traces with the Python SDK: a deterministic trajectory check scored at run time, and a backfill job that grades yesterday's traces.
Step 0: prerequisites
A Langfuse project (cloud or self-hosted) with tracing already flowing - if you are not there yet, the get-started guide covers SDK and OpenTelemetry setups - plus pip install langfuse and the LANGFUSE_* environment variables set.
Step 1: decide what each score means
A score nobody can interpret is noise with a timestamp. Before writing any code, fix three things per score: the name (tool_budget_ok, not quality), the data type, and the exact rule that produces the value. Deterministic rules make the best first scores for agents because trajectories are structured data: tool names, arguments, counts and order can be checked exactly, no judge required.
Step 2: score the run as it happens
Inside instrumented code, score the current trace from the active span. Two boolean checks - a tool budget and an ordering rule - look like this:
from langfuse import Langfuse
langfuse = Langfuse() # reads LANGFUSE_* env vars
with langfuse.start_as_current_observation(as_type="span", name="handle-ticket") as span:
result = run_agent(ticket)
span.score_trace(
name="tool_budget_ok",
value=1 if len(result.tool_calls) <= 8 else 0,
data_type="BOOLEAN",
)
span.score_trace(
name="lookup_before_refund",
value=1 if looked_up_before_refund(result.tool_calls) else 0,
data_type="BOOLEAN",
)
The grader itself is ordinary code:
def looked_up_before_refund(tool_calls):
names = [c["name"] for c in tool_calls]
if "issue_refund" not in names:
return True # nothing irreversible happened
refund_at = names.index("issue_refund")
return "lookup_order" in names[:refund_at]
Keep run-time graders cheap and side-effect free; anything slow or model-graded belongs in the offline path below.
Step 3: backfill traces after the fact
Scores do not have to be written by the code that produced the trace. Anything that knows a trace's id can grade it later - a nightly job, a human review tool, an eval runner. The low-level call takes the trace id directly:
langfuse.create_score(
trace_id=trace_id,
name="lookup_before_refund",
value=1,
data_type="BOOLEAN",
comment="lookup_order preceded issue_refund",
)
A practical backfill job has three parts: pull yesterday's traces through the SDK's API client or an export, extract each trace's tool calls from its observations, and write one create_score per trace with the grader above. The scores documentation covers the full parameter set, including observation_id for scoring a single step and idempotency for safe re-runs - re-grading after a grader fix should overwrite, not duplicate.
Start the backfill on a small window, spot-check a handful of graded traces by hand, and only then widen it. A grader bug applied to a month of traces is a month of wrong dashboards.
Step 4: read scores as distributions, not lights
One green number invites complacency. Useful readings are comparative: pass rate this week against last, scores segmented by prompt version or model, the slice of traces where tool_budget_ok fails clustered by input type. When a score moves and no code changed, suspect the grader or the traffic before the agent.
Two scores will not tell you whether the agent is ready to ship - that takes a curated dataset, calibrated judges and CI gating, which is eval suite engineering territory, and any model-graded score you add on top needs the calibration treatment first. But two honest deterministic scores on live traffic beat a dashboard of unmeasured ones, and you can have them running today.