Stage 6: Evaluation and Feedback

A diagnostic reference. Read while optimizing an existing retrieval-augmented system. Each section serves either a human diagnosing the system or an agent interrogating it.

Concept

Evaluation and Feedback closes the loop. Every earlier stage transforms a query or corpus. This stage observes what users did with the results and produces two artifacts: a quality measurement and training material for the next ranking version. The operational sequence is query, results, actions, logging, aggregation or learning, then a measured ranking update.

The stage sits last in serving order and first in influence. Its inputs come from the serving path (the query as issued, the result list as shown with its ordering, and identifiers sufficient to join them) and from the application UI (every attributable user action: clicks, add-to-carts, purchases, skips, dwell, thumbs, returns). Its outputs flow back upstream: aggregated per-(query, document) boosts and weighted judgment lists to Ranking & Fusion (stage 4), log-mined vocabulary (related queries, misspellings, acronym expansions) to Query Understanding (stage 2), and experiment verdicts to the whole pipeline. Generation-stage interfaces (stage 5) produce behavioral signals of their own (copy, thumbs, regeneration requests), and the same capture schema applies to them.

The source course organizes the consumption side into four categories: signals boosting (per-(query, doc) boosts; head queries only), learning to rank (a model over query-document features; all queries), collaborative filtering (personalized affinities), and knowledge graph learning (domain vocabulary from logs). What carries the stage: signal collection is identical across all four; only the consumption varies. Get the capture schema right once and every category becomes an aggregation or training job over the same substrate. The teaching schema is five columns (query_id, timestamp, user_id, type, target), with two design decisions that carry everything and cannot be retrofitted: query_id is a per-instance ID rather than a query-text key, and the full results list is logged rather than only the clicks. Its production form is User Behavior Insights (UBI), the emerging industry standard: same shape, more detail.

One further discipline distinguishes this stage from a metrics dashboard: the training set derived from behavior is itself a model, with its own error modes, and must be validated like one. Judgments carry known biases (position, sparsity, presentation), and each correction has a name and a technique in the catalog below.

What good looks like at this stage: every (query, results, action) tuple is logged with a per-instance query ID; judgments derived from behavior are corrected for position bias and shrunk toward a prior where evidence is thin; changed judgments are A/B tested as a signals override before any model trains on them; and a deliberate exploration budget exists, because a system that only learns from what it already shows can only become a sharper copy of itself.

Symptom catalog

The catalog is ordered by dependency rather than frequency. Capture defects come first because they are unrecoverable retroactively and gate everything else; judgment-construction biases follow; process-discipline failures close. The featured symptom is presentation_bias_echo_chamber, this stage’s analog of retrieval#recall_metric_ambiguity: the misunderstanding that silently invalidates the rest of the toolbox.

Three framing-level symptoms declared in stage 0 (framing#no_feedback_loop, framing#no_labeled_queries, framing#proxy_metric_optimization) route their fixes to this chapter. They have no catalog entries here because they are diagnosed at stage 0; their intervention rows open the hypothesis map below.

missing_results_list: Only clicks are logged

• Observable signal: The signal store records user actions but nowhere records the result lists users actually saw, or their ordering.
• Frequency: Very common. Teams instrument the action and forget the context.
• Cost of leaving unaddressed: Two channels are lost at once. Skips become invisible (a user who saw [A, B, C] and clicked A and C passed over B; one skip is noise, skips in aggregate are negative evidence), and position-bias correction becomes impossible, because it requires knowing what was shown and in what order. The loss is permanent for all traffic logged before the fix.

query_id_as_text: Signals keyed on query text

• Observable signal: The join key between queries, results, and actions is the query string or a hash of it, rather than a fresh per-instance ID minted at each visit to the search bar.
• Frequency: Common. The instance-ID semantics are non-obvious and text keys feel natural.
• Cost of leaving unaddressed: Two users running “iPad”, or one user running it twice, collapse into one record. Result sets may have differed between the instances, so actions attribute to results the user never saw, corrupting every downstream judgment.

no_negative_signals: Only the positive channel collected

• Observable signal: The signal taxonomy contains clicks, add-to-carts, and purchases, and nothing that subtracts: no skips, no remove-from-carts, no returns.
• Frequency: Common. Blog-post treatments of signals boosting cover only the positive channel, and negative signals require the results list plus deeper instrumentation.
• Cost of leaving unaddressed: Rejected results keep whatever boost they earned. A document that was briefly popular and is now actively avoided stays pinned near the top on accumulated positive signal.

session_attribution_gap: Sessions cannot be reconstructed

• Observable signal: Session-scoped computations (skip mining, per-session judgments) are impossible because the raw signals lack the identifiers post-processing needs: query_id, user_id, timestamps.
• Frequency: Common in systems that log events into analytics tools designed for page views rather than search attribution.
• Cost of leaving unaddressed: Session boundaries are not knowable in real time; the correct design logs raw signals as they come and derives sessions later. Without the identifiers, that later derivation has nothing to join on, and the judgment-construction techniques in this chapter cannot run.

zero_click_session_loss: The click model drops sessions without clicks

• Observable signal: The judgment pipeline processes only sessions containing at least one click. Nobody has measured the zero-click fraction of traffic.
• Frequency: Common wherever SDBN-style click models are used, since the examine boundary is defined by the last click.
• Cost of leaving unaddressed: Proportional to the zero-click share. Where that share is high, most of the evidence is silently discarded, and the judgments describe a biased minority of traffic. Mitigations: measure the share first; then either accept the loss or substitute softer pre-click events (hover, scroll-into-view, dwell, partial play) as examine or interest evidence. UI design is itself a sparsity intervention: a stickier, more interactive results page manufactures more attributable evidence per session.

position_bias_unaccounted: Judgments are raw CTR

• Observable signal: Implicit judgments are computed as clicks over impressions per (query, document). A CTR-by-rank breakdown, if plotted, decays in a power-law shape independent of content.
• Frequency: Very common. CTR is the obvious first computation and its bias is invisible without the per-rank plot.
• Cost of leaving unaddressed: Whatever the legacy ranker put on top looks relevant, because users scan top to bottom and click what they see. The lesson’s lab artifact: “A-Team” and “Fast Five” earn anomalous CTR for a “Transformers Dark of the Moon” query purely because the legacy ranker placed them high. The correction is to replace impressions with examines: everything at or above the last-clicked rank in a session counts as examined, everything below is excluded from the denominator. This computation, SDBN (Simplified Dynamic Bayesian Network), reaches roughly 90% of the full DBN’s accuracy at a fraction of the complexity. High-ranked items collect examines whether or not they earn clicks, so their CTR stops running away on position alone; a result clicked at rank 19 earns a strong ratio precisely because it was rarely examined. The field has a name, click models, and a free textbook (Click Models for Web Search); most practitioners reinvent pieces of it without knowing it exists. The ranking-side view of this failure is ranking_and_fusion#position_bias_labels; the two IDs describe one defect observed at different stages.

sparse_judgment_overconfidence: One click, grade 1.0

• Observable signal: Tail (query, doc) pairs with one click on one examine carry the same grade as pairs with a hundred clicks on a hundred examines, and outrank better-evidenced pairs in the judgment list.
• Frequency: Very common in any judgment pipeline built on raw division.
• Cost of leaving unaddressed: The tail of the judgment list is dominated by flukes, and the trained ranker learns from them with full confidence. The fix is Bayesian: start every pair at a Beta-distribution prior (the lesson’s illustration: alpha 15, beta 20, a default grade near 0.43, deliberately below 0.5, innocent until proven relevant). Clicked examines increment alpha, unclicked examines increment beta; the grade is the posterior mean, and variance shrinks as evidence accumulates. Soft labels reach the ranker as sample weights of roughly 1/variance passed to LambdaMART or XGBoost, so confident pairs count more in training; in pairwise LTR the two examples’ variances add. The ranking-side view is ranking_and_fusion#sparse_judgment_overconfidence.

stale_feature_aggregation: The window outlives the cause

• Observable signal: Judgments aggregated over a long window (a month or more) reward documents whose clicks came from a transient state: on sale at the time, since discontinued, briefly in the news.
• Frequency: Common wherever aggregation windows are chosen for volume without regard to feature volatility.
• Cost of leaving unaddressed: The click’s cause was a feature state, and aggregation erases the timestamp of that state. Labels reward documents for reasons that no longer hold, and the trained model inherits the confusion. The labels-as-signals A/B test (see judgments_never_ab_tested) catches this cheaply before it becomes a training artifact.

deep_funnel_label_drift: Conversions as labels import confounders

• Observable signal: Labels are built from deep-funnel events (purchase, checkout, signup). Search “wins” experiments while downstream business metrics fall, or the reverse; cheap, easily purchased items climb; browsing-rich queries look like failures.
• Frequency: Common in organizations where revenue events are the only well-instrumented telemetry.
• Cost of leaving unaddressed: Deep-funnel events attribute weakly to the search and import confounders unrelated to relevance: price, checkout friction, inventory. The attribution principle is closeness to search: the click is the middle point, pre-click events attribute strongly, funnel events weakly. The structural resolution is to let click-adjacent events build the label and use deeper events as confirmations or guardrails: a click with dwell or an upvote counts, a click with no positive downstream is treated as if it did not happen.

personalization_query_collapse: The judgment key omits user state

• Observable signal: In a personalized system, judgments keyed on query text alone misrank for individual users; a “jumper” judgment built from all users fails the user who only ever sees size large.
• Frequency: Specific to systems with meaningful personalization or user-scoped filtering.
• Cost of leaving unaddressed: Distinct intents conflate under one key, and the judgment list averages over populations that never see the same results. Redefining the query as (text, user-state) restores fidelity at the price of sparsity: every disaggregation buys correctness with data volume, and the Bayesian prior machinery above becomes more load-bearing as keys get thinner.

presentation_bias_echo_chamber: The model learns the ranker that fed it

• Observable signal: Whole regions of feature space contain no training examples at all. The lesson’s running case: no examples with both high title BM25 and high overview BM25, because the legacy ranker never sent users there. New or differently shaped candidates never accumulate evidence; popularity compounds (popular items earn signal, signal earns rank, rank earns signal).
• Frequency: Universal in behaviorally trained rankers that lack an exploration mechanism. This is the stage’s featured symptom.
• Cost of leaving unaddressed: Training examples are biased toward how the current algorithm works, because that is what users were shown. Users can only click what they see; if the ranker never shows a class of documents, that feature region collects no evidence, and the trained model reproduces the ranker that generated its data. If the search is bad, CTR-driven training promotes more bad things. Position-bias corrections cannot reach this; SDBN fixes attribution within the shown list, and this failure is about what was never shown.

The escape is active learning: spend a small slice of live traffic gathering evidence about under-explored regions. Fit a Gaussian process over observed (feature vector, grade, variance) examples; it interpolates predicted grade and inflates variance away from observations. Score candidates with an acquisition function (the source course teaches probability of improvement, where high predicted grade combined with high variance wins), with a theta parameter dialing exploration up or down. The deployment shape is a two-path architecture: an exploit path (production LTR on a minimal proven feature set) and an explore path (Gaussian process over a feature superset, injecting candidates at roughly result slot 3 or interleaved into the page). Explore-path clickstream enriches the dataset; once a new feature has coverage, the exploit path adopts it. A useful reframe: an LTR system is in some ways a feature exploration system, and any new feature idea creates demand for clickstream in the region that feature distinguishes. The ranking-side view is ranking_and_fusion#presentation_bias_echo_chamber.

judgments_never_ab_tested: Labels treated as ground truth

• Observable signal: Click-model output flows directly into model training. No intermediate deployment validates the labels themselves.
• Frequency: Very common. Teams treat labels as ground truth rather than as a model with its own error modes.
• Cost of leaving unaddressed: Label bugs (click-model defects, stale-feature aggregation) surface weeks later as inexplicable trained-model behavior, where they are expensive to trace. The discipline: before training, deploy the judgments as a signals override (literally pin the click model’s preferred results for the affected queries) and A/B test that. If the labels are wrong, the failure surfaces on a cheap, instantly reversible override. The lesson’s illustration: memorize the right answers for two head queries and show them to users in an A/B test.

How to inspect

By code pattern

Greppable signals in the capture and judgment pipeline.

Pattern / location	What you’re looking for	What suggests a problem
Signal schema definition (event tables, tracking calls)	The five fields or their UBI equivalents: query_id, timestamp, user_id, type, target	Missing results-type rows (suggests missing_results_list); missing user_id or timestamps (suggests session_attribution_gap)
The query_id assignment site	A fresh UUID per search-bar visit	query_id = hash(query_text) or the raw text as key (suggests query_id_as_text)
Signal type enum or taxonomy	Negative types: skip, remove_from_cart, return	Positive-only taxonomy (suggests no_negative_signals)
Aggregation SQL (GROUP BY keys)	user_id in the dedup key; normalized query text	No per-user dedup (spam exposure); raw query text fragmenting head queries into many keys
Signal weight constants	Type weights in the spirit of click 1, add-to-cart 10, purchase 25	All types weighted equally, or funnel events mixed into a composite label (suggests deep_funnel_label_drift)
Decay terms (0.5 ** (age / half_life), EXP(...) on age)	Half-life decay matched to corpus tempo	No decay at all; or a half-life orders of magnitude off the domain (hours for news, weeks for jobs, months for retail)
Judgment computation	Examine-based denominator (SDBN)	clicks / impressions with no rank logic (suggests position_bias_unaccounted)
Prior constants (alpha, beta, smoothing terms)	A Beta prior with mean below 0.5; sample weights near 1/variance	Raw division with no prior (suggests sparse_judgment_overconfidence); hard labels discarding variance
Aggregation window bounds	Window sized to feature volatility	Month-plus windows on volatile catalogs (suggests stale_feature_aggregation)
Exploration code (acquisition functions, interleaving, bandit arms, GP fitting)	Any deliberate explore path	Complete absence of exploration code in a behaviorally trained ranker (suggests presentation_bias_echo_chamber)
Training-data pipeline entry points	A labels-as-signals override step gated by an A/B test	Click-model output piped straight to the trainer (suggests judgments_never_ab_tested)

By measurement

Metrics to compute over the signal store before trusting any judgment built from it.

Measurement	How to compute	Healthy range / red-flag threshold
CTR by rank	Per rank position, clicks over impressions across all queries	A raw power-law decay that judgment construction ignores is the signature of position_bias_unaccounted
Zero-click session share	Fraction of sessions with no click event	Domain-dependent; if large and the click model drops those sessions, zero_click_session_loss is in effect
Per-user signal concentration	Distribution of signals per (user, query, doc); top-contributor share	A few users dominating a pair’s signal suggests spam or missing dedup
Judgment evidence distribution	Histogram of examine counts behind each judgment	A large mass at 1 to 2 examines with grades of exactly 0.0 or 1.0 indicates sparse_judgment_overconfidence
Feature-space coverage of training data	Plot or bucket training examples over pairs of ranking features	Empty regions (for example, no examples high on two features at once) are the concrete signature of presentation_bias_echo_chamber
Results-list completeness	Fraction of click events joinable to a logged results row	Below 100% means partial missing_results_list; also a cheap observability probe for silent logging regressions
Boost staleness	Age distribution of the signals behind current boosts	Mass older than the domain’s natural tempo suggests missing or mis-tuned decay
Label-vs-experiment agreement	For judgment changes deployed as signals overrides, the A/B verdict	Overrides that lose their A/B test mean the labels are wrong; this measurement existing at all is the point

Diagnostic questions

Triage flow for an agent or engineer assessing this stage. Earlier questions are coarse; branches go deeper when answers are unclear or red-flag.

1. [interview] Does the system learn from user behavior at all? If no, this is framing#no_feedback_loop; the intervention is the first row of the map below, and every subsequent question is premature.

2. [code] What does one logged search look like in the data? Ask for an actual row set. Branches:

   • No results-list row: missing_results_list. Fix capture before anything else; historical traffic is lost.
   • Join key is query text: query_id_as_text.
   • No user_id or timestamps: session_attribution_gap.

3. [code] What action types are collected, and do any subtract? Branches:

   • Positive-only: no_negative_signals.
   • Funnel events (purchase, signup) mixed into a weighted label: probe for deep_funnel_label_drift; ask whether deep events could instead guard the click.

4. [interview] How does a click become a judgment? Branches:

   • Raw CTR over impressions: position_bias_unaccounted; ask for the CTR-by-rank plot.
   • Examine-based: ask how examines are defined (above last click is the SDBN convention) and proceed.

5. [code] What happens to a (query, doc) pair with one click on one examine? Branches:

   • Grade 1.0: sparse_judgment_overconfidence; ask whether a prior and sample weights would reach the trainer.
   • Prior present: ask whether variance flows to training as sample weights or is discarded at labeling.

6. [code] What fraction of sessions have zero clicks, and where do they go? Branches:

   • Unknown: measure first; the answer decides whether zero_click_session_loss matters here.
   • Known and large, sessions dropped: discuss softer pre-click events and UI instrumentation.

7. [interview] Are the labels ever deployed and tested before a model trains on them? Branches:

   • No: judgments_never_ab_tested; the signals-override test is cheap and reversible.
   • Yes: ask what the last such test caught; a test that has never failed anything may not be real.

8. [code] How long is the aggregation window, and what changes about a document inside that window? Branches:

   • Long window, volatile catalog (sales, stock, news cycles): stale_feature_aggregation.

9. [code] Does any traffic deliberately explore? Branches:

   • No exploration anywhere: presentation_bias_echo_chamber is active by construction; ask for the feature-space coverage plot to make it concrete.
   • Exploration exists: ask where explore results are placed (a mid-page slot or interleaving, never the whole page) and how theta or its equivalent was chosen.

10. [interview] Is the system personalized, and what is the judgment key? Branches:

    • Personalized, text-only key: personalization_query_collapse; discuss (text, user-state) keys and the sparsity cost.

Interventions by likely cause

Ordered by frequency of cause, except the first three rows, which receive the framing chapter’s hand-offs: framing#no_feedback_loop, framing#no_labeled_queries, and framing#proxy_metric_optimization all route their fixes here, so an engineer arriving from chapter 0 lands on a next action immediately.

Hypothesis	Supporting evidence	Intervention	Expected impact	Cost / risk
No feedback loop exists at all (framing#no_feedback_loop)	The same query returns the same wrong results indefinitely; quality plateaued long ago	Stand up the five-column signal schema (query_id as instance ID, results list logged, negative types included); consume it first as signals boosting at stage 4	Removes the structural cap on quality; lesson 1’s claim is that no algorithm change can substitute	Medium: instrumentation touches the UI and serving path; the schema itself is small
No labeled queries to measure against (framing#no_labeled_queries)	No file or table maps queries to known-good results; changes ship on visual inspection	Derive implicit judgments from the signal store via examines and Bayesian priors; where signals are also absent, start with a small hand-judged set (tens of queries) as the bridge	Converts relevance from opinion to measurement; unblocks every stage’s “what good looks like”	Low to medium: the click-model path needs signals first; the hand-judged bridge costs an afternoon
Team optimizes proxies instead of relevance (framing#proxy_metric_optimization)	Reported metrics are benchmarks, engagement counts, latency; none measures relevance for actual users	Adopt behavior-derived judgments as the reported quality metric; validate them with the labels-as-signals A/B discipline so the proxy critique does not simply recurse	Optimization effort re-aims at user-recognizable relevance	Low: measurement and reporting work once signals exist
Result lists were never logged (missing_results_list)	Click events cannot be joined to a shown-results row	Add a results-type row per search, carrying the ordered doc IDs; backfill is impossible, so ship it now	Unlocks skip mining, negative signals, and all position-bias correction for future traffic	Low: one event type; the cost is only the delay in shipping it
Signals keyed on query text (query_id_as_text)	Distinct search instances collapse into one record	Mint a UUID per search-bar visit; propagate it through results and action rows	Actions attribute to the result set the user actually saw	Low: a key change plus propagation
Judgments are raw CTR (position_bias_unaccounted)	CTR-by-rank decays as a power law; legacy top results look uniformly relevant	Replace impressions with examines (SDBN): denominator is everything at or above the last click	Removes the dominant bias in implicit judgments at low complexity (roughly 90% of full DBN accuracy)	Low: a post-processing change over existing signals, given the results list exists
No prior, no shrinkage on sparse pairs (sparse_judgment_overconfidence)	One-examine pairs carry grade 1.0	Beta prior (mean below 0.5) per (query, doc); grade as posterior mean; sample weights near 1/variance passed to the trainer	Tail judgments stop dominating on flukes; training respects confidence	Low: arithmetic on the aggregation job
Labels flow to training unvalidated (judgments_never_ab_tested)	No signals-override step between click model and trainer	Deploy changed judgments as a pinned signals override for affected queries; A/B test that before training	Label bugs and stale-feature artifacts surface on a cheap, instantly reversible surface	Low: reuses existing boost and experiment machinery
No exploration budget (presentation_bias_echo_chamber)	Feature-space regions with zero training examples; popularity compounds; new candidates never surface	Add an explore path: Gaussian process over a feature superset, probability-of-improvement acquisition, candidates injected at roughly slot 3 or interleaved; keep theta small and nonzero	The training set gains coverage where the ranker was blind; new features become adoptable by the exploit path	Medium: exploration spends real result slots on real users; the budget is a product decision as much as a modeling one
Zero-click sessions silently dropped (zero_click_session_loss)	Zero-click share is high and the click model requires a click	Measure the share; if large, add softer pre-click events (hover, dwell, scroll-into-view) as examine evidence, or redesign the results UI to elicit more attributable interaction	Recovers evidence from discarded traffic, at some noise cost	Medium: event instrumentation; noisier labels need the prior machinery
Aggregation window outlives the click’s cause (stale_feature_aggregation)	Long windows on a volatile catalog; overrides built from the labels lose their A/B test	Bound the window by feature volatility; rely on the labels-as-signals test as the standing tripwire	Labels stop rewarding expired states	Low: window tuning plus the discipline above
Conversion-only labels (deep_funnel_label_drift)	Cheap items climb; search wins while business metrics sag, or the reverse	Rebuild labels from click-adjacent events; demote deep-funnel events to guardrails (a click with no positive downstream is discarded)	Labels measure relevance rather than price and checkout friction	Low to medium: label pipeline change
Judgment key ignores user state (personalization_query_collapse)	Personalized system; per-query-string judgments misrank per user	Rekey judgments as (query text, user-state segment); accept the sparsity cost and lean on priors	Judgments match what each user population actually sees	Medium: every disaggregation buys correctness with data volume
Signal store silently degrading (cross-cutting: observability)	Results-list completeness or signal volume drops without alarm	Monitor the signal pipeline as a production system: completeness ratios, volume baselines, join-rate alerts	Logging regressions stop destroying the loop invisibly	Low: standard observability applied to an overlooked pipeline