Chapter 6 ended with a trace that can show whether a poor answer began with missing evidence or unsupported synthesis. Now ask the movie search system for “robot falls in love.” Its fused ranking places WALL-E first and Her fourth, and the evaluation set marks both films relevant. A viewer passes WALL-E, opens Her, returns to the results a few seconds later, and rewrites the query.
The viewer’s quick return complicates an otherwise reassuring result. The ranking places both films judged relevant among its first four results, so offline evaluation can show strong agreement with the fixed human judgments. Online evaluation adds what the viewer saw and did after those results appeared. Neither source explains the quick return by itself. Her may have been a poor result, or the viewer may have learned enough from its page to ask for something more specific.
The team must reconstruct the evidence before it can tell those possibilities apart: the judgments used to score the ranking, the films shown to the viewer, the actions that followed, and the trace that produced the result order. Feedback begins only after those observations have a defensible meaning. Otherwise the next version may learn from an incomplete view of the session and a mistaken account of what the viewer wanted.
A metric inherits its judgments
The offline report begins with the “robot falls in love” ranking and human judgments about which films fit that request. Each metric focuses on a different part of the ranking’s agreement with those judgments.
If the team cares about concentration near the top, precision@k measures the fraction of the first k results judged relevant. If it cares about recovering the relevant material it knows about, recall@k measures how many known relevant results appear within that cutoff. Calculating recall also requires a credible estimate of the relevant material beyond the visible list.
Some tasks succeed as soon as the system places one relevant result near the top. Reciprocal rank measures that case by dividing one by the position of the first relevant result. A first-place result scores 1, a second-place result scores 1/2, and a third-place result scores 1/3. Mean reciprocal rank (MRR) averages those scores across a set of queries.
Normalized discounted cumulative gain (nDCG) examines more of the ranked list. It gives more credit to relevant results near the top, where users are more likely to see them. When the judgments distinguish levels of relevance, nDCG also gives more credit to a highly relevant result than to a partly relevant one.
The instrument below makes the dependency visible. Choose “robot falls in love.” The fixed ranking places WALL-E first and Her fourth, with both marked relevant. Change Her to not relevant. Precision@5 falls, recall@10 and reciprocal rank stay at 1, and nDCG@10 rises to 1 because WALL-E is now the only relevant film and already occupies the ideal first position. Because the instrument displays one query at a time, it reports reciprocal rank rather than MRR. Its nDCG calculation uses binary judgments, so every relevant film receives the same gain.
Choose a query, then select any result row to cycle its judgment through unjudged, relevant, and not relevant. The ranking stays fixed, so you can watch which metric values change and which cutoffs explain the response.
The exercise changes the evaluation without moving a film. Precision@5 now counts one relevant film instead of two, while reciprocal rank still finds its first relevant film at rank 1. nDCG also becomes perfect because its ideal ranking now contains only WALL-E, which the system already placed first. The scores disagree about the change because each summarizes a different part of the same judgment set.
Once the judgment set is fixed, the calculation is repeatable. Its meaning still depends on human choices about query coverage, relevance grades, assessor guidance, disagreements, unjudged results, and stale labels. The score for “robot falls in love” describes agreement with a particular judgment version. Carefully reviewed judgments make that comparison useful, but they cannot explain why the viewer opened Her, returned, and rewrote the query.
Reconstruct what the user experienced
The live investigation starts with an impression: the ordered results presented for one search request. The click on Her becomes interpretable only when the team can reconstruct the alternatives, including WALL-E in first place. This execution needs a unique request identifier rather than the query text alone because another search for “robot falls in love” may use a different index, filter, experiment, or ranking version.
The impression first records the results returned by the backend, then distinguishes the ones the interface actually exposed. Pagination, scrolling, collapsed groups, and rendering failures can keep a candidate offscreen. If the viewer saw WALL-E and passed it, that non-click still does not prove the film was irrelevant. Position and presentation shape attention, and the viewer may already know the film or prefer the live-action premise of Her.
Clicks, saves, reformulations, and later outcomes must link back to the impression that preceded them. A play or save of Her may occur well after the viewer leaves the results, yet it would change how the team interprets the quick return. The logging design needs an attribution rule that decides which impression receives credit for that later outcome. Here, the quick return and rewritten query still matter because opening the film page may not have completed the task.
The system should preserve those raw impression and action events so session boundaries, labels, and aggregate metrics can be derived later under versioned rules. It should collect only the user or session identifiers required for the analysis, with appropriate consent, access controls, and retention limits. Recording exposure and outcomes independently of one ranker’s scores keeps the history usable after that ranker changes.
The request identifier also joins the impression to the operational trace inherited from Chapter 6. That trace reconstructs query processing, retrieval, ranking, generation, and any retry loop, including stage timings, model and prompt versions, index version, route decisions, token use, and typed errors. Together, the records show both what the viewer could experience and how the system produced it.
The joined records still do not explain whether Her helped the viewer. The trace may reveal slow retrieval, a changed index, or a retry that altered the ranking, making the session useful for review without turning it into a quality label. Before the team adds that session to a versioned evaluation set, it must remove sensitive material and apply the current rubric. Training use requires separate approval because the viewer’s behavior was shaped by the ranking and interface they received.
Preserve events before interpreting them
The durable event stream should describe the Her interaction without depending on the internal representation of the ranker that served it. Stable query, request, result, user or session, action, and outcome identifiers remain interpretable when the scoring model changes. A proprietary score alone may not because its scale and meaning can disappear with the model that produced it.
Because those identifiers survive a scoring-model change, the same history can support several later claims. Aggregate query-result interactions can become behavioral boosts, while request, result, and outcome tuples can become examples for learning to rank. User-item histories can support collaborative filtering, and reformulation pairs can reveal spelling variants, acronyms, and domain synonyms for query understanding.
Each use interprets the same observation differently, so its derived dataset needs separate, versioned rules. A play label, a query-item boost, and a reformulation-based synonym candidate make different claims about the same movie session. Keeping their transformations explicit lets the team revise an interpretation without rewriting the historical event stream.
The preserved events still describe behavior only for the results and positions chosen by the serving system, leaving the team without observations about films another ranker might have shown. This presentation bias means the old ranking policy influenced which results could collect clicks, so the team must decide what the observed sequence can support before turning it into a training label.
Turn an event sequence into a label hypothesis
The impression records Her entering the viewport, receiving a click, and holding the viewer for only a few seconds before they return and reformulate the request. That sequence provides progressively different evidence: exposure establishes that the film could be examined, the click adds evidence of interest, and the quick return weakens the case that opening the result completed the task. It does not reveal whether the viewer rejected the film, found it unavailable, disliked its detail page, or learned enough to refine an otherwise successful search.
The team can interpret the sequence only after choosing the outcome the ranker should improve. If the target is topical fit, the quick return can qualify the human judgment on Her without erasing it. If the target is helping the viewer choose a film, a later play or save can carry more weight, although the label now reflects availability and page quality alongside ranking.
With the target fixed, the event sequence changes the team’s confidence rather than supplying a certain grade. An examined skip may count as weak negative evidence, a click as a possible positive, and a click followed by a successful outcome as a stronger positive, while an immediate return can weaken that reading. The strength of each interpretation also depends on the attribution window, session boundary, and treatment of missing events, so those rules must remain explicit and versioned.
Applying those rules produces a versioned label dataset whose lineage leads back to the source events and transformation logic. The figure below generalizes the Her session into a ten-film impression and follows the boundary from session rules to a derived label. Independent evidence must then test whether the label’s ordering corresponds to the intended outcome before a model can learn from it.
Test the interpretation before training
Suppose the label rule treats the click on Her followed by a quick return as weak negative evidence. That label is a hypothesis about the session, not a fact supplied by the event stream. The first test asks whether higher and lower labels correspond to the outcome they claim to predict, using evidence that did not create them. Independent human judgments can supply that check, as can outcomes from a later time period. If the same clicks both define and validate the label, the test only confirms its construction.
A later sample of movie sessions tests whether the rule survives beyond the period that produced it. If Her becomes unavailable in one region, for example, quick returns there may rise even though the film still fits the query. An overall average could hide that pattern, so the team should compare movie-query types, result positions, device layouts, and regions, then review representative sessions to see which condition changed the label’s meaning.
Once the label points in the right direction, replay or shadow evaluation can test the ordering it implies before training. The team can first apply the label as a temporary feature that lowers Her for similar requests, then inspect whether the resulting order is actually better. In a shadow run, the candidate system scores live requests without changing what users see. If the direct ordering is poor, a learned model will usually spread the same problem across many weights and make its source harder to isolate.
A guarded A/B test can then compare the current ranking with the version that lowers Her for similar requests. Before assigning traffic, the team should define the primary outcome, such as a later play or save, along with guardrails for reformulation and abandonment, the attribution window, and the stopping rule. This comparison shows whether the proposed interpretation improves live movie-search outcomes rather than merely producing a plausible new order.
The same sequence applies if the system uses the ranked summaries to generate an answer to “robot falls in love.” Hold the request, retrieved film summaries, structured output contract, and evaluation rubric fixed, then vary one prompt or model candidate at a time. Per-case results remain necessary because a better average can hide a refusal regression or a class of claims unsupported by those summaries. A system prompt earns release through measured behavior on representative and adversarial movie requests, not through a persuasive reading of its wording.
Labels that pass these checks remain bounded by the impressions that produced them. They can describe the exposed results, including WALL-E and Her, but they cannot describe a film buried below the first page. Closing that gap requires the system to collect some evidence beyond its existing choices.
Collect evidence beyond the current ranking
Blade Runner sits below the first page for “robot falls in love,” so neither the existing impression nor the clicks on WALL-E and Her reveal whether viewers would accept it as a result for this request. Meanwhile, films already near the top keep collecting interactions, those interactions strengthen their labels, and the next model becomes more confident in the same choices.
An exploration budget uses a small, controlled share of exposure for candidates whose value remains uncertain. This creates the exploration-exploitation tradeoff: exploitation serves the strongest known choice, while exploration gathers evidence that may reveal a better one.
Before exploring Blade Runner, the system must still enforce regional availability, subscription permissions, audience filters, and minimum quality. The policy can place an eligible explored film only in a lower-risk position, cap its share of traffic, monitor plays, saves, and reformulations, and stop automatically if those outcomes deteriorate.
When the policy exposes Blade Runner, the impression log must mark it as explored and record its propensity, the probability that the policy assigned to that display. Propensity makes the sampling process inspectable and supports later corrections such as inverse propensity weighting, which accounts for the fact that some candidates had a greater chance of appearing than others. Without that probability, randomized traffic creates observations while hiding how they were sampled.
The movie system can spend this budget by placing an eligible film such as Blade Runner in a randomized lower slot or by using an adaptive policy that changes its exposure as evidence accumulates. In either case, the team must cap the explored traffic, identify every affected impression, reverse the policy when needed, and compare what it learns with any decline in plays or saves, or any rise in reformulations.
The “robot falls in love” request can now be followed from the judgments on WALL-E and Her through the viewer’s impression, quick return, label rules, outcome checks, and the controlled exposure of Blade Runner. Together, those records make quality and uncertainty observable without confusing an event with its interpretation. Chapter 8 moves that evidence inside the request path, where an agent can choose whether to answer, search again, or stop. The agent gains flexibility, but its choices remain only as reliable as the measurement path established here.