Chapter 6 ended with two responsible ways to respond: answer from the available evidence or state that the evidence is insufficient. Neither response proves that the system helped the user. A grounded answer can still miss the request, while a refusal can signal appropriate caution or a failure to retrieve an available source.
Evaluation asks whether those responses satisfy the definition of relevance established in Chapter 1. Offline evaluation measures performance on a fixed set of queries and judgments, making system versions easy to compare before release. Its conclusions extend only as far as that test set.
Online evaluation observes what happens after release: whether users engage with results, reformulate queries, complete tasks, or leave. These signals cover real requests, but they are also shaped by presentation, latency, availability, and other factors beyond relevance. Offline tests isolate a controlled comparison; online evidence shows whether that comparison predicted the user experience.
Feedback turns those observations into changes upstream. Judgments, impression logs, and outcomes can inform query processing, ranking features, training labels, and release decisions. If logging is incomplete or labels are biased, the next version will optimize a distorted account of what users saw and did.
The goal of this chapter is to determine whether a change improved the user’s outcome and to identify which stage produced the difference.
Metrics depend on human judgments
Offline evaluation begins with a ranked result list and a set of relevance judgments for the query. The metric does not observe relevance directly. It summarizes how the ranking compares with those judgments, so its meaning depends on which documents were judged and how relevance was defined.
Precision@k measures the fraction of the first k results judged relevant. Recall@k measures how many of all known relevant documents appear within that cutoff. Recall therefore requires a credible estimate of what relevant material exists beyond the visible results.
Reciprocal rank is the inverse of the position of the first relevant result. Mean reciprocal rank (MRR) averages that value across a query set. nDCG, normalized discounted cumulative gain, evaluates more of the ranking by rewarding relevant results near the top and, when judgments have grades, giving more credit to higher relevance.
The instrument below holds one ranking fixed while you change its judgments. Because it displays one query at a time, it reports reciprocal rank rather than MRR. Its nDCG calculation uses binary judgments, relevant or not relevant, so every relevant document receives the same gain.
Choose a query, then select any result row to cycle its judgment: unjudged, relevant, then not relevant.
The ranking stays fixed. Watch which metric values change and which cutoffs explain the response.
Unjudged and not relevant both receive zero gain in this binary example. Recall uses every document currently marked relevant as its known relevant set.
Changing one judgment may move several metrics, one metric, or none, depending on the document’s position and each metric’s cutoff. The ranking has not changed. The instrument has changed the evidence used to evaluate it.
The calculation is repeatable once the judgment set is fixed. The judgment set still carries human decisions about query coverage, relevance grades, assessor guidance, disagreements, unjudged documents, and when labels become stale. A report that says “nDCG improved” is incomplete without the query set and judgment version against which it improved.
Carefully reviewed judgments provide a strong offline baseline, but they are expensive to produce and cover only selected queries. The next source of evidence is behavior from live traffic, which offers scale only when the system records what users actually had a chance to see.
Log complete impressions and outcomes
To interpret online behavior, the system must record which results the user had an opportunity to select. A click is meaningful only when the system can reconstruct what the user could have chosen instead. An impression records the results presented for one search request, including their order and enough context to identify the system that produced them.
Each request therefore needs a unique identifier for that execution rather than the query text alone. Two searches for the same words may run against different indexes, filters, experiments, or ranking versions. Clicks, saves, reformulations, and later outcomes must link back to the particular impression that preceded them.
The impression should distinguish results returned by the backend from results the interface actually exposed. Pagination, scrolling, collapsed groups, and rendering failures can prevent a candidate from being seen. A non-click on an exposed result is useful evidence in aggregate, but it is not automatically a negative judgment. Position and presentation affect attention, and the user may have stopped after finding an acceptable result earlier in the list.
Outcomes may occur after the search interaction. A purchase, completed workflow, support escalation, or successful answer can arrive minutes or days later, so the logging design needs an attribution rule that defines which request receives credit. Query reformulation and abandonment also matter because they can reveal that a click did not complete the task.
Preserve the raw impression and action events so session boundaries, labels, and aggregate metrics can be derived later under versioned rules. Collect only the user or session identifiers required for that analysis, with appropriate consent, access controls, and retention limits. Once exposure and outcomes are recorded independently of a particular ranker’s scores, the same evidence can evaluate more than one ranking system.
Operational traces belong beside this event stream, but they answer a different question. A trace reconstructs one execution through query processing, retrieval, ranking, generation, and any retry loop. It records stage timings, model and prompt versions, index version, route decisions, token use, and typed errors under the same request identifier used by the impression. The impression explains what the user could experience; the trace explains how the system produced it.
That distinction protects later evaluation. A slow, corrected, abandoned, or high-cost request is a useful candidate for review, but the trace is not a quality label by itself. Sample those cases, remove sensitive material, and apply the current rubric before promoting them into a versioned evaluation set. Training data needs a separate approval because live behavior contains presentation bias, abuse, and policy choices made by the serving system.
Reuse behavior data across ranking systems
An impression log should describe the interaction without depending on one ranker’s internal representation. Stable query, request, result, user or session, action, and outcome identifiers remain interpretable when the scoring model changes. A proprietary score alone does not because its scale and meaning may disappear with the model that produced it.
Different consumers can then derive different evidence from the same events. Aggregate query-result interactions can support behavioral boosts. Request, result, and outcome tuples can become examples for learning to rank. User-item histories can support collaborative filtering, while reformulation pairs can reveal spelling variants, acronyms, and domain synonyms for query understanding.
Those derived datasets should remain separate from the raw log and carry their own versioned rules. A purchase label, a query-item boost, and a synonym candidate make different claims about the same interaction. Keeping the transformation explicit lets a team revise one interpretation without rewriting the historical event stream.
Reuse does not remove the presentation bias identified in Chapter 5. The log contains behavior only for the results and positions chosen by the serving system, never for the documents another ranker might have shown. The same events can inform a new model, but they cannot be treated as neutral evidence that the previous ranking policy had no influence on what users clicked. Turning behavior into training labels therefore requires another layer of care.
Convert behavior into training labels carefully
Presentation bias is only one reason raw clicks cannot serve directly as relevance grades. Position affects which results are examined, and sparse observations carry substantial uncertainty. Converting behavior into a training label adds another decision: what does each recorded action say about relevance?
Exposure signals such as scrolling a result into view or hovering over it can help estimate whether the user examined the result. They provide little evidence that the result was useful. A click indicates attraction or intent, but it can still be accidental, exploratory, or driven by presentation rather than relevance.
Actions after the click can strengthen or qualify that signal. Dwell, saving, adding to a cart, completing a workflow, or purchasing may indicate that the result helped. Each later action also introduces other causes. Price, availability, page quality, checkout friction, and the time allowed for conversion can change the outcome after ranking has done its work.
The label rule should begin with the outcome the ranker is meant to improve. If the target is topical relevance, a purchase should confirm evidence from the search interaction rather than receive an arbitrary large weight. If the target is completed purchases, conversion can be the outcome, but the team must acknowledge that the label now includes product and checkout effects as well as ranking quality.
Within that target, the event sequence supplies confidence rather than certainty. An examined skip may be weak negative evidence, a click may be a candidate positive, and a click followed by a successful outcome may support that positive more strongly. Immediate return to the results can weaken the interpretation without proving that the document was irrelevant. The attribution window, session boundary, and treatment of missing events must all be explicit and versioned.
These rules produce a versioned label dataset whose lineage should lead back to the source events and transformation logic. Before the labels influence a model, independent evidence must show that their ordering corresponds to the intended outcome.
Validate labels against outcomes
Behavioral labels encode a hypothesis about the event stream. A click may stand for relevance, a completed task may confirm it, and an examined skip may count against it. Before a ranker learns from those interpretations, the team should test whether higher labels actually correspond to the outcome the label is meant to predict.
The first check uses evidence that did not create the labels. Compare label levels with independent human judgments or with outcomes from a later time period. If the same clicks define and validate a label, the test can only confirm its own construction. A temporal holdout also reveals labels that captured a promotion, inventory state, or seasonal preference that no longer applies.
Aggregate agreement can hide local failures, so validation should also inspect query types, positions, devices, markets, and other segments that materially change the interaction. A label can correlate with the desired outcome overall while reversing direction for an important slice of traffic. Qualitative review of representative queries remains useful because it exposes patterns an average conceals.
The derived labels can then be applied as a temporary feature or boost in replay or shadow evaluation, where the candidate system scores live requests without changing what users see. This tests the ordering the labels imply before a learned model distributes their influence across many weights. If the direct ordering is poor, training on it will usually make the source of the problem harder to isolate.
A guarded A/B test provides the final check against live outcomes. The experiment should define its primary outcome, guardrail metrics, attribution window, and stopping rule before traffic is assigned. The amount of experimental machinery should match the risk of the change, but every release still needs a comparison that can distinguish improvement from ordinary variation.
Prompt and model changes need the same controlled comparison. Hold the question, retrieved evidence, structured output contract, and evaluation rubric fixed, then vary one prompt or model candidate at a time. Record per-case results as well as the mean, because an improved average can conceal a refusal regression or an unsupported-claim class. A system prompt earns release through measured behavior on representative and adversarial cases. A persuasive reading of its wording supplies no such evidence.
Labels that pass these checks remain limited by the exposure that produced them. They can validate behavior on observed results, but they cannot describe candidates the current ranker never showed. Closing that gap requires the system to collect some evidence outside its existing policy.
Reserve traffic for exploration
This missing coverage is the presentation bias identified in Chapter 5. Correcting position bias does not create evidence for unseen documents. Without new exposure, highly ranked items collect more interactions, those interactions strengthen their labels, and the next model becomes more confident in the same choices.
An exploration budget assigns a small, controlled share of exposure to candidates whose value remains uncertain. This is the exploration-exploitation tradeoff: exploitation serves the strongest known choice, while exploration gathers evidence that may reveal a better one.
Exploration should operate inside the same eligibility constraints as normal ranking. Permissions, safety rules, availability, required filters, and minimum quality thresholds still apply. The policy can limit exploration to lower-risk positions, cap the traffic share, monitor guardrail metrics, and stop automatically when user outcomes deteriorate.
The log must identify explored impressions and record the probability that the policy assigned to each displayed candidate, its propensity. That probability makes the exposure process inspectable and supports later corrections such as inverse propensity weighting for the fact that some candidates had a greater chance of being shown than others. Without it, randomized traffic produces new observations but leaves the sampling process hidden.
Different products can spend the budget by inserting an eligible candidate into a randomized slot or by using an adaptive policy that shifts exposure as evidence accumulates. The mechanism matters less than preserving the contract: exploration is bounded, attributable, reversible, and evaluated against both learning value and user cost.
The chapter began with fixed judgments and offline metrics, then followed live behavior through impressions, reusable events, proxy labels, outcome validation, and controlled exploration. The loop can now change the system without confusing observations with judgments or treating feedback as neutral. Chapter 8 moves the next decision inside the request path, where an agent can inspect results and choose another action. That flexibility still depends on the measurement discipline established here.