The Film Club asks for one film about artificial intelligence and one about memory loss, a comparison of how each treats identity, and a recommendation for a group that prefers ambiguity over action. One run verifies Ex Machina and Eternal Sunshine of the Spotless Mind, recommends Eternal Sunshine after four calls, and keeps the shared-watchlist change behind its approval boundary. Another makes nearly the same recommendation after twelve calls, repeated searches, a lost supporting passage, and an attempt to write before approval.
Because the two recommendations look alike, the evaluator has to inspect the trajectory, the complete record of the actions, evidence, decisions, and stopping condition behind each answer. Chapter 11 produced that record by keeping the model’s proposal, the application’s policy decision, the bounded approval, and the watchlist service result separate, giving the evaluator evidence to judge each stage and the route as a whole.
Judge the result and the route
The difference between the two runs appears at three nested units, each locating a failure that the others cannot explain on its own.
| Unit | Core question | Example measures |
|---|---|---|
| Step | Was this action valid and useful? | tool selection, arguments, retrieval relevance, policy, latency |
| Trajectory | Did the sequence make safe and efficient progress? | completion, duplicate work, retries, handoff loss, cost |
| Outcome | Did the result help the user and remain supported? | task success, factuality, citation support, preference fit |
Trajectory B’s premature watchlist write fails when that individual action is judged against the approval rule. Removing the call would repair the policy violation, but the run would still contain twelve calls for work that Trajectory A completed in four. That waste appears only when the evaluator widens its view from one step to the complete trajectory, and even an efficient route can still end at the wrong film. The final outcome must be judged in turn to establish whether the recommendation answers the request and remains supported.
No single score can replace those judgments. A rubric states how the evaluator combines them into a verdict, but the underlying results must remain available so an engineer can trace a regression to retrieval, planning, state, policy, synthesis, or resource use.
The instrument below holds the two trajectories fixed. Under “Compare trajectories,” change the evaluation lens and watch the verdict move, then inspect the events behind each score. Open “Calibrate judge” and switch from “Answer only” to “Trace + evidence” to see how the available record changes what a reviewer can establish.
Same request · Similar final answer
Compare two films about identity, recommend the more ambiguous one, and add it to the shared watchlist only after approval.
Choose what counts
Score the final recommendation and ignore how it was produced.
Both paths pass
Score comparison
The answer-only lens passes both paths because their recommendations are nearly identical. The balanced rubric rejects Trajectory B after path, safety, and resource judgments enter the score, and the safety-first lens widens the gap. Those verdicts expose a difference the answer concealed, but locating the repair requires the trace beneath the score.
Read the trace before the score
Trajectory B’s failing score answers the release question, but it does not tell an engineer what to change. Reopening the trace shows that a supporting passage was available earlier but missing when the comparison reached synthesis, which locates that defect in evidence handling after retrieval. Earlier events account for the excess calls: the route queried an unavailable source and repeated searches without adding evidence. The later watchlist attempt belongs to the approval boundary, so it remains a separate repair even though it contributed to the same failing score.
Locating those failures depends on preserving the connections between events. Source identifiers and validation results establish which passages entered the record, while state transitions show whether that evidence survived until synthesis. The policy decision and approval state remain attached to the watchlist call, and timing and cost stay with every span. Without those links, the trace would show activity without revealing where verified evidence vanished or why the write could not proceed.
An evaluator can use these links to explain one failed run. Monitoring groups the same failures across live traffic while preserving the events behind each result. If missing-support failures rise after a prompt release, observability lets the team reopen the affected traces and see whether retrieval failed or evidence disappeared later.
Latency exposes another reason to preserve the route. The artificial-intelligence and memory-loss branches run at the same time, so adding their durations counts part of the wait twice. The request’s elapsed time follows the critical path, the chain of dependent work that finishes last.
The spans in the figure add up to 3.25 seconds, but the request finishes in 2.6 because the artificial-intelligence branch completes while the memory-loss branch is still running. Reconstructing that difference requires start and end times for queue waits, retrieval and model calls, validation, retries, and the points where branches split and rejoin, all tied to one request identifier. For a streamed response, the trace also separates time to first token from the time when the complete response arrives.
One trace explains this 2.6-second request; a distribution shows whether the delay is typical. The median, or p50, marks the midpoint, while p95 or p99 exposes the slow tail. If that tail grows, route, model, prompt, and index versions identify which system choice changed, while cache state, query class, attempt count, and terminal status describe the conditions under which it became expensive. Those slices can reveal a small-model route that saves money on most requests but creates retries that dominate the tail.
Turn the route into test cases
The trace lets the team replay the two Film Club runs after a system change and ask whether the same failures return. A passing case can recommend Eternal Sunshine of the Spotless Mind or another choice supported by the request, provided the path preserves two verified film records and keeps the shared write behind approval.
Trajectory B determines which parts can be checked in code. Tool schemas and call limits expose invalid or excessive actions, required approvals guard the watchlist boundary, and source and citation identifiers establish whether the expected evidence survived. Those deterministic checks cannot decide by themselves whether the comparison explains the films accurately or whether the recommendation fits the group’s preference. Those properties still need semantic judgments about relevance, support, and preference fit.
The expected outcome must also allow valid alternatives. A strong evaluator should accept a different film when its evidence satisfies the request rather than penalizing every answer that differs from one reference. Ordinary successes, missing search branches hidden by fluent prose, correct answers after duplicate work, and safe pauses before a shared write can then cover different failure locations without forcing one exact trajectory.
These cases express product decisions about which outcomes, paths, risks, and resource limits count. Version the tasks, traces, rubrics, and automated judges together so a changed score can be attributed to a system change or to the evaluation machinery itself.
Calibrate model judges against people
Semantic judgments are harder to encode as fixed rules. A language model can compare the recommendation with the request, assess whether a passage supports a claim, or explain a preference mismatch, which makes model judges useful for running a judgment set repeatedly.
The generator and judge may still share the same blind spots. In the instrument above, open “Calibrate judge” and compare the answer-only view with “Trace + evidence.” The model judge passes the plausible recommendation, while the human reviewer cannot verify every comparison claim from the answer alone. Once the trace exposes the lost passage, the missing support becomes a concrete failure.
Calibration measures model judgments against decisions from people who understand the task. A disagreement can reveal vague criteria, missing evidence, an unstable judge prompt, or a case that belongs in deterministic code. It can also reveal a rubric that rejects a valid alternative because it has learned one reference answer too narrowly.
Once the rubric separates fixed rules from semantic judgments, some checks can run at much larger scale. Microsoft Research’s Agent-Pex extracts behavioral rules from prompts and traces, checks argument validity, output compliance, and plan sufficiency, and can invert those rules into adversarial tests. Its reported evaluation of more than 5,000 Tau2 traces shows how specification-based trace inspection can scale, but those checks still cannot decide on their own whether the Film Club comparison is well supported.
Self-RAG places critique inside the run itself: a trained model decides when to retrieve and critiques its passages and generations. That mechanism can improve the path before an answer is returned, but it does not give evaluation an independent judge. Self-RAG’s benchmark results support the retrieval-and-critique method rather than a claim that model self-critique can replace human calibration.
Test changes offline, then watch live traffic
Calibration makes the model judge usable, but a comparison still has to isolate the change being tested. For the Film Club case, hold the two retrieved records and the structured response contract fixed while changing the system prompt or model. Code can check the response status, citations, approval state, latency, and cost, while the calibrated judge reads the comparison for support and preference fit. Preserving the prompt and model version with the result makes a failure reproducible instead of leaving retrieval drift as another possible cause.
Uber’s Cart Assistant uses this same division in a production shopping system. Its evaluation set combines designed requests with edge cases drawn from anonymized production traffic. Code checks whether the draft cart contains valid items, quantities, and propagated constraints, while a human-calibrated multimodal judge assesses whether the cart and response satisfy the shopper’s request. When a candidate scores worse than the baseline, engineers inspect the step-level traces to decide whether the product lost a constraint or the judge misunderstood a valid result. The Film Club test asks the same question when a passage disappears or a supported alternative receives a failing score.
Those regression cases cover failures the team knows how to ask about before release. Live traffic adds requests and operating conditions the set did not anticipate. Monitoring can separate runs that ended in a tool error or policy pause from completed runs with unsupported claims or user corrections. The trace then shows whether retries, latency, and token use made an otherwise successful answer too expensive.
Bayer and Thoughtworks’ PRINCE system joins those two evaluation loops. It reruns a curated dataset after changes to the workflow, prompts, or models, then evaluates live traffic each day. Because PRINCE retains intermediate workflow stages and exposes tool use and citations, a drop in answer or context quality can be traced to the evidence and action that produced it. The Film Club system needs the same return path: a known lost-passage case can block a release, while an unseen failure from live traffic can become the next regression case after review.
That review matters because production traffic contains ambiguous requests, policy-sensitive data, and outcomes shaped by the current system. A live trace can supply a test case, but it should not become an automatic training label.
Make coordination earn its cost
If the memory-loss branch is assigned to a specialized agent, its evidence must return with source identifiers and request constraints intact before the comparison can begin. A valid search that drops its citation weakens the combined trajectory, while a second agent repeating the same search adds cost without improving coverage.
Delegation may still earn that cost when it expands evidence coverage or isolates access to a tool. Each handoff also copies context across another authority boundary and adds latency and another place for the run to fail, so the delegated route should be compared with a single-agent or fixed-workflow baseline on the same tasks. Agent count remains an architectural choice rather than a measure of quality.
Close the loop without hiding the path
Whether one workflow or several agents ran the branches, Trajectory B can now fail even though its recommendation resembles Trajectory A’s. The trace exposes its wasted work, missing support, and premature write attempt, which turns the verdict into a diagnosis. The judgment set makes those defects repeatable, while calibrated judges and production monitoring show whether a later change repairs them. Planning, recovery, and authorization have all left evidence in the same record, allowing evaluation to close Part II’s control loop without hiding how the result was produced.
Both Film Club trajectories still searched one shared body of evidence. If the signed-in principal changes which records are eligible, evaluation begins too late unless authorization has already shaped retrieval. Chapter 13 opens Part III by following that decision through lexical and dense candidate generation before restricted evidence can reach ranking or generation.