Your offline suite is green on 180 cases and you are still nervous. That is the correct reaction. An offline dataset is a sample of the traffic you thought of, frozen at the moment you curated it. Production sends inputs nobody wrote a case for: half-finished sentences, a pasted email thread, an order id that does not exist, the same customer asking the same thing four times.
Shadow mode closes that gap without putting the new version in front of anyone. You mirror real requests to the candidate, throw its answers away, and score both versions on the same inputs. Then a canary sends a small slice of real traffic to the candidate under guardrails. This tutorial builds both, plus the rule that decides when to cut over.
What shadow mode does and does not buy you
It buys you: real input distribution, real trajectory lengths, real tool latency and cost, and a paired comparison on identical inputs. It does not buy you: any signal about how users respond to the new answers, because they never see them. Click-through, escalation rate and thumbs-down only move once real traffic is on the candidate, which is what the canary step is for.
It also costs money. Every shadowed request runs the agent twice. Sample it.
Step 0: the two hard prerequisites
Read-only tools, or no shadow run. The shadow agent must not send email, issue refunds, write to the CRM or charge anyone. Route it at a tool layer where every write is either stubbed or hard-blocked. If your tool client cannot enforce that, build the enforcement before you build the shadow runner. A shadow refund is a real refund.
BLOCKED = {"issue_refund", "send_email", "update_account", "create_ticket"}
def shadow_tool_client(name, args):
if name in BLOCKED:
return {"status": "blocked_in_shadow", "tool": name, "arguments": args}
return real_tool_client(name, args)
Returning a visible blocked_in_shadow marker beats raising: the trajectory still shows the agent tried the write, which is exactly the thing you want to compare between versions.
Never block the user. The shadow call runs out of band, on a queue or a background task, with its own timeout and its own error budget. If the shadow path can add latency to the live response or fail the request, it will, at the worst possible time.
Step 1: mirror a sample of requests
Capture the request at the same boundary your live agent uses, tag it with a run id, and hand it off.
import random, uuid
SHADOW_RATE = 0.05 # 5% of traffic
async def handle_request(req):
pair_id = str(uuid.uuid4())
live = await run_agent(req, version="v1", pair_id=pair_id)
if random.random() < SHADOW_RATE:
# fire and forget; never awaited on the request path
enqueue_shadow(req.snapshot(), pair_id)
return live.answer
Two details matter more than they look.
req.snapshot() must capture everything the agent reads: user message, conversation history, tenant and user id, feature flags, retrieved context if retrieval happens upstream. A shadow run on a partial input compares two different questions.
pair_id is what makes this a paired comparison. Both runs carry it as a trace attribute, so you can join them later and diff case by case instead of comparing two aggregate averages. Paired analysis needs far fewer samples to see the same effect, which matters when you are paying twice per request.
The shadow worker itself is unremarkable:
def shadow_worker(snapshot, pair_id):
try:
run_agent(
snapshot,
version="v2-candidate",
pair_id=pair_id,
tool_client=shadow_tool_client,
timeout_s=90,
)
except Exception as exc:
record_shadow_error(pair_id, exc) # errors are data, not alerts
Shadow errors are a result. If the candidate times out on 3% of real inputs and never did offline, that is the finding.
Step 2: score both runs online
Online scoring is a different job from offline scoring: no reference answer exists, so every grader has to be reference-free. In practice that means deterministic trajectory checks first, judges only where structure cannot answer the question.
If your traces are in Langfuse, attach scores to each trace with the version and pair id as attributes, the same way as in scoring agent traces in Langfuse:
from langfuse import Langfuse
langfuse = Langfuse()
def score_run(trace_id, run):
names = [c["name"] for c in run.tool_calls]
checks = {
"completed": int(run.finished and run.answer is not None),
"tool_budget_ok": int(len(names) <= 8),
"no_repeat_loop": int(max_consecutive_repeat(names) <= 2),
"schema_valid": int(answer_parses(run.answer)),
"attempted_write": int(any(n in BLOCKED for n in names)),
}
for name, value in checks.items():
langfuse.create_score(
trace_id=trace_id, name=name, value=value, data_type="BOOLEAN"
)
Five booleans, no model in the loop, and they already catch the failures that hurt most on rollout day: the agent that stalls, the agent that loops, the agent that emits unparseable output, the agent that suddenly wants to write.
On top of that, sample maybe 200 pairs for a model-graded comparison. Reference-free grading is best done pairwise — show the judge the input and both trajectories, ask which one handled it better, and randomize which is A — because relative judgments are steadier than absolute scores on a rubric nobody validated. Then measure the judge against human labels on a subset before you believe any of it; the procedure is in an unmeasured LLM-as-judge is just a guess. A pairwise judge that agrees with your reviewers 62% of the time is a coin with opinions.
Record the boring operational numbers per version too: p50 and p95 end-to-end latency, tool calls per run, tokens per run, cost per run, error rate. Half the cutover decisions we see turn on cost and latency, not quality.
Step 3: read the diff, not the average
Join on pair_id and put every pair in one of four buckets:
| Bucket | Meaning | What to do |
|---|---|---|
| both pass | no change | ignore |
| both fail | pre-existing failure | backlog, not a blocker |
| v1 pass, v2 fail | regression | read every one of these by hand |
| v1 fail, v2 pass | fix | confirm it is a real fix, not a lucky guess |
Only the two disagreement buckets carry information, and they are usually small enough to read. A net pass-rate move of +2 points can hide 40 regressions and 60 fixes, which is a very different system, not a slightly better one. Cluster the regressions by input type before deciding anything: five regressions all on multi-account users is a bug with a name, five scattered singletons is noise.
And it may well be noise. On paired data the useful test is over the disagreements only: with b cases where v1 passed and v2 failed and c the reverse, a McNemar-style check asks whether b and c differ more than a coin flip would explain. With b=12 and c=18 you have nothing. With b=3 and c=40 you have something. Same reasoning as measuring noise in agent pass rates, applied to pairs.
Every regression you accept as real should leave the shadow run as a case in the offline suite. That is the whole point of doing this on live traffic: it hands you the cases you could not invent, and once they are in the dataset with recorded tool responses they gate the next change too.
Step 4: the canary rule, written down before you look
Shadow mode ends where user-visible behavior begins. The canary is where you find out whether people escalate more, retry more, or quietly give up.
Write the rule before you have the data, and write it as numbers:
- Slice: 5% of traffic, held for at least 24 hours, covering a full daily cycle. Route by stable user hash, not per request, or a user gets two different agents mid-conversation.
- Exclusions: name the tenants and flows that never get canaried — the enterprise account in a renewal, anything touching payments.
- Guardrails, checked every 15 minutes: error rate, p95 latency, human-escalation rate, cost per conversation, and the
attempted_writeandno_repeat_loopscores from above. Each gets a threshold and an automatic revert. - Promotion criteria: the guardrails held, the paired disagreement test favors the candidate or is flat, and every regression cluster has been read and either fixed or accepted in writing.
- Revert path: one flag flip, tested before the canary starts, with someone actually watching for the first hour.
The half of this that teams skip is the exclusion list and the revert drill. A revert path nobody has exercised is a hope.
What this does not tell you
Shadow mode measures the candidate on the traffic you get today. It says nothing about inputs an attacker will send tomorrow — indirect injection through tool output, authorization probes, exfiltration paths — because those are not in your traffic yet. Those need deliberate adversarial cases run as their own suite. It also will not fix a thin offline dataset; it will only show you how thin it was.
What it does give you is the thing that is hard to get any other way: evidence about the new version on real inputs, before a single user sees it. If you want help standing up the shadow runner, the online scorers and the cutover rule on your stack, that is eval suite engineering and ongoing reliability engineering work — or start with the contact form and describe the agent, its tools and the deadline.