Stage 4: Ranking and Fusion

A diagnostic reference. Read while optimizing an existing retrieval-augmented system. Each section serves either a human diagnosing the system or an agent interrogating it.

Concept

The ranking and fusion stage takes the candidate set produced by retrieval and reorders it into the list the user (or the generation stage) actually sees. Retrieval optimizes for recall at tolerable cost; this stage optimizes for precision at the top of the list, using scorers that are too expensive or too data-hungry to run over the whole corpus. It is the last stage that reorders: everything before it widens or shapes the candidate pool, everything after it consumes the ordering.

The stage’s organizing frame is the cost-ordered phase pipeline. Ranking is a sequence of phases, each rescoring a strictly smaller candidate set, ordered cheapest first: a cheap candidate phase (BM25, quantized ANN) narrows the corpus to hundreds or thousands; a middle phase (learning-to-rank, signals boosts, filters) narrows to the low hundreds; the most expensive phase (cross-encoder, full-precision vector rescoring, or an LLM) touches tens. The rule is simple: never run an expensive scorer on a candidate set a cheaper scorer could have narrowed first. Cost means compute, latency, and memory together; they usually move as one. A corollary that matters diagnostically: each phase’s depth caps what later phases can fix. A cross-encoder reranking the top 10 cannot rescue a relevant document at rank 40.

Write a candidate budget beside every phase. Counts such as 1,000 retrieved, 100 feature-scored, 20 cross-encoder reranked, and 5 shown are useful starting hypotheses rather than portable defaults. On a labeled set, record where relevant documents land before each phase and set the windows to cover that distribution. A later model cannot rescue a candidate an earlier window excluded.

The same compatibility rule applies to ranking features. Construct historical training rows from the values available when the impression happened rather than from today’s price, stock, popularity, or user profile. Training on current values leaks information the serving ranker could not have known. The safest path is one versioned transformation contract shared by feature logging, offline training, and online scoring.

Three scorer families fill those phases, and the source course prescribes an explicit order for adopting them: memorize before you generalize. Signals boosting memorizes right answers for head queries by aggregating user behavior per (query, document) pair into an additive boost. Learning to rank generalizes to the tail by training a model (LambdaMART on gradient-boosted trees is the modern classic) over a small hand-engineered feature set, using judgments typically derived from those same clicks. Cross-encoder reranking supplies the semantic final phase where query-document similarity dominates relevance. Getting signals boosting solid on head queries first, verifying it in A/B tests, then training LTR for the tail is the prescribed sequence; skipping straight to LTR is the named anti-pattern. The stage also carries a second sense of fusion beyond the retriever-output fusion documented at Stage 3: composing heterogeneous score sources (base relevance, boost, model score) into one final ordering without letting the same evidence count twice.

What good looks like at this stage: the top of the list is right for the queries users actually run, at a latency the product can afford, and the training loop that produces the ranking does not simply reflect the previous ranking back at itself.

Symptom catalog

The catalog leads with judgment-construction failures. They silently corrupt everything downstream, and they are the least likely to be suspected because the symptoms surface as “the model isn’t good enough” rather than “the labels are wrong.”

presentation_bias_echo_chamber: The ranker trains on its own output

• Observable signal: Click-trained ranking keeps promoting the same class of results; new or differently-shaped candidates never gain ground; offline metrics look fine because they are computed on the same biased labels.
• Frequency: Universal in click-trained systems without an exploration mechanism. A matter of degree rather than presence.
• Cost of leaving unaddressed: Whole regions of feature space stay dark forever. If the search is bad, click-trained ranking promotes more of the same badness, with offline metrics reporting success throughout.

Training data only contains what the current ranker showed. If the ranker never surfaces a class of documents (say, high overview-field BM25 with low title-field BM25), users cannot click them, so no evidence for that region ever enters the judgments. Position-bias corrections do not reach this; they debias clicks among shown results, and this failure is about results never shown. The escape is exploration: fit a Gaussian process on (features, grade, variance) triples so unobserved feature regions get a predicted grade and a predicted uncertainty, score candidates with an acquisition function (high predicted grade with high variance is the sweet spot), and inject winners into live traffic to harvest clicks the production ranker would never have generated. The mature form is a two-path architecture: an exploit path (production LTR on a minimal proven feature set) and an explore path (a Gaussian process over a feature superset) feeding observations back. Full treatment of the judgment-construction side lives at evaluation_and_feedback#presentation_bias_echo_chamber.

position_bias_labels: Judgments inherit the legacy ranker’s ordering

• Observable signal: Click-derived judgments strongly correlate with the old system’s ranking. Items the legacy ranker put on top look relevant regardless of merit; the “best” documents for a query are suspiciously often the ones that were already ranked first.
• Frequency: Very common. Raw CTR (clicks over impressions) is the naive baseline nearly every team starts from.
• Cost of leaving unaddressed: The new model learns to imitate the old ranker. Training spend produces a ranker that reproduces the system it was meant to replace.

Users click what they see, and they see the top. Raw clicks over raw impressions rewards inherited rank. The fix is the examines correction: replace impressions with examines, defined as everything at or above the last click in a session. Items below the last click are treated as unseen and leave the denominator, so a result clicked at rank 19 earns a strong ratio precisely because it was rarely examined. This is the Simplified Dynamic Bayesian Network (SDBN), which captures roughly 90% of the full DBN’s accuracy at a fraction of the complexity. Two boundary conditions to check while inspecting: the session definition (one query and the actions on that result page; a query change starts a new session) and zero-click sessions, which are invisible to SDBN and must be dropped or backfilled with softer pre-click events (hover, dwell, scroll). Full treatment at evaluation_and_feedback#position_bias_unaccounted, the same defect viewed from the judgment pipeline.

sparse_judgment_overconfidence: Raw ratios carry no confidence

• Observable signal: Rarely-shown documents with one or two lucky clicks outrank consistently-validated ones. The judgment table contains grades of 1.0 sitting on a denominator of 1.
• Frequency: Very common wherever judgments come from clicks and the query distribution has any tail.
• Cost of leaving unaddressed: The training set treats a single lucky click and a hundred consistent clicks as identical evidence. Model quality degrades exactly where data is thinnest.

One click on one examine and 100 clicks on 100 examines both divide to 1.0 with wildly different confidence. The fix is a Beta prior: seed each (query, document) with pseudo-counts (the lesson’s illustration is alpha 15, beta 20, a deliberate below-0.5 default, innocent until proven relevant), increment alpha with clicked-examines and beta with unclicked-examines. The posterior mean becomes the grade; the posterior variance becomes the confidence, entering LambdaMART or XGBoost as sample weights of roughly 1/variance. The lessons give no procedure for choosing the prior constants; tune them to signal volume and treat the published numbers as an illustration. Full treatment at evaluation_and_feedback#sparse_judgment_overconfidence.

popularity_feature_loop: The model rediscovers its own label source

• Observable signal: Feature importance shows popularity dominant and semantic similarity negligible. Rankings drift toward pure popularity ordering over successive retraining rounds.
• Frequency: Common in any LTR system that both derives labels from clicks and feeds click-derived popularity in as a feature.
• Cost of leaving unaddressed: A self-reinforcing loop: popularity drives clicks, clicks drive labels, labels train popularity importance. The tail gets ranked by a signal that has no information there.

This is a within-model cousin of the echo chamber. Some teams accept the loop deliberately for head queries: strong signals carry the head, and the model’s non-popularity features carry the tail. If the loop is unintentional, the diagnostic move is feature ablation (train on random feature subsets and compare) followed by a deliberate composition rule; the full mitigation is the exploration machinery under presentation_bias_echo_chamber.

dynamic_feature_leakage: Features computed at training time rather than click time

• Observable signal: Model performs well offline, poorly online, and the gap concentrates in queries where volatile features (price, availability, promotion badges) changed between the click and the training run.
• Frequency: Common wherever features are joined onto judgments from a live table at training time.
• Cost of leaving unaddressed: Labels are attached to a state of the world that no longer exists. The model learns relationships that were never true at decision time.

The fix is feature logging discipline: for dynamic features, log the feature value at click time and train on the logged value. Engines support returning per-feature scores in the search request itself (OpenSearch/Elasticsearch named queries with include_named_queries_score, Solr pseudofields, Vespa computed fields), which makes click-time logging a one-request pattern rather than a separate pipeline.

query_split_leakage: Train/test split at the wrong level

• Observable signal: Suspiciously strong offline metrics that evaporate in A/B tests.
• Frequency: Common in first LTR implementations; the default train_test_split over rows produces it silently.
• Cost of leaving unaddressed: Offline evaluation is meaningless. Every modeling decision made against it is unvalidated.

Feature vectors in LTR are query-dependent: the same document has different features under different queries, and rows from one query are strongly correlated. A document-level (row-level) split puts rows from the same query on both sides, and the model is graded partly on queries it saw in training. Split at the query level, always.

signal_spam: One user moves a head query

• Observable signal: A product or document climbs a head query’s results, and the signal behind the climb traces to one or a few user IDs.
• Frequency: Common in any signals-boosted system without per-user deduplication; guaranteed eventually in adversarial domains (marketplaces, anywhere sellers benefit from rank).
• Cost of leaving unaddressed: Head queries, the exact queries signals boosting exists to protect, become manipulable at the cost of one obsessive or adversarial user.

The fix is layered rather than absolute. First, deduplicate per user: add user_id to the aggregation’s GROUP BY and count one signal per user per (query, document), which raises the attack cost from one account clicking repeatedly to thousands of fake accounts. Second, weight purchases and other paid actions above clicks, which raises the attack cost to real money.

fragmented_signals: Query variants split their evidence

• Observable signal: Boost tables full of near-duplicate keys (“iPad”, “ipad”, “Ipad”, “iPad 2”), each with weak counts. Boosting underperforms on queries that are obviously popular.
• Frequency: Very common; it is the first defect in any naive aggregation.
• Cost of leaving unaddressed: Signal that should pool into a strong boost fragments into many weak ones, none of which clears the noise floor. The head of the distribution behaves like the tail.

The fix is query normalization inside the aggregation: lowercase, collapse whitespace, optionally stem, before grouping. This is the first of lesson 3’s four refinements to the naive GROUP BY, and the cheapest.

stale_signals: Yesterday’s winners never age out

• Observable signal: Last year’s or last season’s winners still rank on time-dated queries. Users searching for the holiday schedule get the old holiday schedule.
• Frequency: Common; decay is the refinement teams most often skip because the failure takes months to appear.
• Cost of leaving unaddressed: Boosts calcify. The system’s memory of past relevance overrides present relevance exactly on the queries where users notice.

The fix is half-life decay in the aggregation: weight * 0.5^(age / half_life). The half-life is domain-specific by orders of magnitude (hours for news, roughly a month for job boards, weeks to months for e-commerce), so a copied constant can be as harmful as no decay. Done right, decay plus fresh clicks makes time-dated head queries self-correct: the new year’s document overtakes the old one with zero engineering intervention.

flat_signal_weights: All signals count the same

• Observable signal: Boost values dominated by abundant low-confidence clicks; purchases and other strong signals barely move rankings; negative signals (skips, cart removals, returns) are absent entirely.
• Frequency: Common in first implementations that count events rather than weighting them.
• Cost of leaving unaddressed: The boost measures attention rather than satisfaction. Clickbait-shaped results win over results people actually wanted.

Weight signal types by confidence in what they say about relevance. The canonical starting ratios from lesson 3: click 1, add-to-cart 10, purchase 25, with negative weights for skips, removals, and returns. Weights are set by domain intuition plus qualitative review more than by exhaustive search. Note the boundary with judgment construction: weighted sums are fine for boost tables, but when building training labels, deeper-funnel events work better as confirmations or guardrails on the click than as weighted addends, because the further an event sits from the search action, the more confounders it picks up.

signals_long_tail_no_op: Boosting judged outside its scope

• Observable signal: Team reports “signals boosting doesn’t work.” On inspection, the measurement was over the whole query distribution, or the corpus is transient (marketplace stock-of-one listings, job postings, news) so per-document signal never accumulates.
• Frequency: Common as a conclusion; the underlying scope limit is universal.
• Cost of leaving unaddressed: A working technique gets ripped out, or worse, effort goes into tuning a lever that is structurally incapable of reaching the problem.

Signals boosting works only at the head of the query distribution and only for stable documents. Long-tail queries never accumulate enough signal; transient corpora never let signal accumulate per document. This is a structural limit rather than a tuning failure. The interventions: measure boosting on head queries specifically; for the tail, LTR is the tool that generalizes past the limit; for transient corpora, cluster short-lived items into stable abstract IDs where possible so signal can accumulate at the cluster level.

signals_features_collision: Click data enters the score twice

• Observable signal: Signals boost and LTR model disagree; rankings oscillate between retraining rounds, or the model appears to fight manual boosts.
• Frequency: Common once a system has both a boost layer and a learned model, which is exactly the state the memorize-then-generalize path leads to.
• Cost of leaving unaddressed: Double-counted evidence and unstable composition. Debugging any single ranking becomes guesswork because two components encode the same facts with different lags.

When click-derived signals feed both an additive boost and an LTR feature, the model and the boost encode the same CTR-derived information and can fight each other. Pick a composition rule deliberately. The pragmatic pattern: let strong signals carry head queries directly and let the model’s non-popularity features carry the tail, either by feeding signals in as a feature and accepting the loop knowingly, or by routing LTR only to queries without signal coverage.

rerank_depth_mistuned: Phase windows chosen by copy-paste

• Observable signal: Either relevant candidates sit just below the rerank window and never surface (too shallow), or latency budgets blow up on the expensive phase (too deep, or the cost ordering is inverted and an expensive scorer runs over a set a cheaper phase should have narrowed).
• Frequency: Common; depths are usually copied from a tutorial or a paper and never revisited.
• Cost of leaving unaddressed: Shallow windows put a hard ceiling on ranking quality that no model improvement can lift; deep windows spend the latency budget where it buys nothing.

The lessons supply worked depths as starting points: roughly the top 1,000 to 1,500 candidates pulled into an external LTR service for the middle phase, the top 50 to 100 fed to a cross-encoder for the expensive phase, and 2x to 10x over-fetch for quantized retrieval feeding full-precision rescoring (mechanics at Stage 3, retrieval#underfetch_quantized). The diagnostic that replaces copy-paste: measure where relevant candidates actually land in the pre-rerank ordering, and set each window to cover that distribution. If relevant documents routinely land at rank 150 out of the candidate phase, a top-100 cross-encoder window loses them by construction.

Cross-referenced symptoms (owned by Stage 3)

Two cross-encoder failure modes present at this stage but are anchored in the retrieval chapter, where the retrieval-plus-rerank path is inspected:

• retrieval#crossencoder_training_mismatch: an off-the-shelf cross-encoder trained on general web QA reranks the target corpus worse than the baseline.
• retrieval#crossencoder_recency_blind: cross-encoders score semantic relevance only; newer equivalents rank below older content wherever recency carries relevance.

Stage 4 carries the intervention side of both: a learning-to-rank layer that takes the cross-encoder score as one feature among several (recency, popularity, price), or a post-hoc reweighting applied after cross-encoder scoring. Tree-based LTR wins where ranking depends on statistics; cross-encoders win where ranking is dominated by semantic similarity; the standard bridge is feeding embedding cosine similarity into LTR as one feature.

How to inspect

By code pattern

Greppable signals that reveal ranking-stage configuration choices.

Pattern / location	What you’re looking for	What suggests a problem
Signals aggregation job (SQL GROUP BY over clickstream, or equivalent)	The four refinements: query normalization, user dedup, type weights, decay	Any refinement missing maps to a symptom: no lower()/trim() suggests fragmented_signals; no user_id in the GROUP BY suggests signal_spam; no per-type weights suggests flat_signal_weights; no decay term suggests stale_signals
0.5 **, half_life, decay in aggregation code	The half-life constant	Absent entirely; or a constant copied across domains (a news system with a 30-day half-life, or vice versa)
clicks / impressions, ctr in judgment construction	The position-bias treatment	Raw CTR with no examines logic suggests position_bias_labels; look for last-click session truncation to confirm SDBN is present
alpha, beta, prior constants near grade computation; sample_weight in the training call	Sparsity handling	Grades computed as raw ratios with no prior suggests sparse_judgment_overconfidence; priors present but sample weights unused wastes the variance information
train_test_split over judgment rows	Split level	Row-level or document-level split suggests query_split_leakage; correct code groups by query ID first
Feature joins in the training pipeline (JOIN ... ON current_, live price/stock lookups)	When dynamic features are captured	Features joined from live tables at training time suggests dynamic_feature_leakage; healthy pipelines log feature values at click time
include_named_queries_score, Solr pseudofields, Vespa computed fields	Feature logging mechanism	Absent in a system with dynamic features; features recomputed offline instead
rank:pairwise, lambdarank, xgboost, lightgbm objective config	LTR model class	Pointwise regression objective in production (pairwise almost always outperforms pointwise for ranking)
Rerank window constants (rescore, window_size, top_n into LTR service or cross-encoder)	Phase depths	Values matching a tutorial exactly, never validated against where relevant candidates land pre-rerank; suggests rerank_depth_mistuned
Feature logging and training joins	Point-in-time construction of volatile features	Live price, inventory, popularity, or user features joined at training time rather than the values available when the impression occurred
Order of scoring phases in the query path	Cost ordering	An expensive scorer (cross-encoder, LLM) running before or instead of a cheap narrowing phase
Signals appearing both in a boost clause and in the LTR feature list	Score composition	Click-derived data entering the final score twice suggests signals_features_collision
Boost application point: sidecar lookup at query time vs boost fields in the index mapping	Query-time vs index-time boosting	Either is valid; query-time plus deep paging is a known pain point, index-time means every aggregation triggers re-indexing
Feature list size and contents	Classic LTR feature engineering	Dozens-plus features without ablation; popularity-type features present alongside click-derived labels suggests popularity_feature_loop

By measurement

Metrics, logs, and traces to enable for ranking diagnosis.

Measurement	How to compute	Healthy range / red-flag threshold
Judgment-to-legacy-rank correlation	Correlate click-derived grades with the position the legacy ranker served each (query, doc) at	Strong positive correlation suggests position_bias_labels; after SDBN it should weaken visibly
Judgment denominator distribution	Histogram of examine counts behind each grade	A large mass of grades on denominators below ~10 with no prior or sample weighting suggests sparse_judgment_overconfidence
Offline nDCG vs online A/B delta	Compare offline eval gain against the shipped A/B result, per model version	Repeated large offline-online gaps suggest query_split_leakage or dynamic_feature_leakage; localize by checking whether the gap concentrates in volatile-feature queries
Feature importance report	Standard gain/split importance from the trained trees, per retraining round	Popularity dominant and similarity negligible suggests popularity_feature_loop; track drift across rounds
Boost key cardinality and near-duplicate rate	Count boost-table keys; cluster case/whitespace variants	High near-duplicate rate suggests fragmented_signals
Per-user signal concentration	For each boosted (query, doc), max share of signal from one user	Any (query, doc) where one user contributes a dominant share suggests signal_spam
Signal age distribution	Histogram of signal event ages weighted by their current boost contribution	Old events still carrying large weight on time-dated queries suggests stale_signals
Boost lift, head vs tail	Measure boosting’s ranking lift separately on head queries and tail queries	Boosting judged on the whole distribution understates it; near-zero lift on the head is the real red flag, near-zero on the tail is expected (signals_long_tail_no_op)
Pre-rerank rank of relevant candidates	For a labeled query set, record where relevant docs land in the candidate ordering before each rerank phase	Relevant candidates routinely landing below a phase’s window suggests rerank_depth_mistuned
Result-class diversity over retraining rounds	Track the distribution of served results across feature-space regions (or simpler proxies: source field, document class) per round	Monotonic narrowing suggests presentation_bias_echo_chamber
Per-phase latency	p50/p95 per scoring phase in traces	The expensive phase dominating while operating on an un-narrowed set indicates inverted cost ordering

Diagnostic questions

Triage flow for an agent or engineer assessing this stage. Label provenance comes first: no model question is interpretable until the origin of the grades is established.

1. [interview] Where do the ranking judgments come from? Human-labeled, click-derived, or none (no learned ranking). Branches:

   • Click-derived: continue to questions 2 through 4; they are the highest-yield checks in the stage.
   • Human-labeled: ask about coverage and refresh cadence, then skip to question 5.
   • No learned ranking: skip to questions 5 and 8; this stage reduces to signals hygiene and phase depths.

2. [interview] Were the labels themselves ever A/B tested? The training set is itself a model. Deploying the click-derived judgments directly as a signals override (pin the top-graded results for their queries) and A/B testing that is the stage’s cheapest high-value move. Branches:

   • Never: recommend it before any further model work; if the labels are wrong, this catches it on a cheap rollback instead of inside a trained model.
   • Tested and won: label quality has evidence; move on.

3. [code] How is position bias handled in judgment construction? Branches:

   • Raw clicks over raw impressions: position_bias_labels; introduce SDBN examines.
   • SDBN present: check the session definition (query change starts a new session) and how zero-click sessions are handled.

4. [code] How are sparse judgments handled? Branches:

   • Raw ratios, no prior: sparse_judgment_overconfidence; introduce a Beta prior and variance-derived sample weights.
   • Prior present: ask whether the variance feeds sample weights or is computed and discarded.

5. [code] What does the final score compose, and does click-derived data enter more than once? Branches:

   • Boost and model both consume clicks: signals_features_collision; establish a composition rule (signals carry head, model carries tail).
   • Single entry point: confirm the composition is applied in cost order.

6. [code] In the signals aggregation, which of the four refinements are present: query normalization, per-user dedup, type weights, decay? Each absence maps to a symptom (fragmented_signals, signal_spam, flat_signal_weights, stale_signals). If decay is present, ask how the half-life was chosen; a copied constant is a soft red flag.

7. [interview] Where is signals boosting being measured? Branches:

   • Whole query distribution, or a transient corpus: signals_long_tail_no_op; rescope the measurement to head queries, use LTR for the tail.
   • Head queries specifically: the measurement is sound.

8. [code] What are the phase depths, and where do relevant candidates land pre-rerank? Branches:

   • Depths copied, landing distribution unmeasured: rerank_depth_mistuned; measure before tuning.
   • Expensive scorer running over an un-narrowed set: cost-ordering violation; insert or deepen a cheaper phase first.

9. [code] For the LTR pipeline: split level, and feature capture time? Branches:

   • Document-level split: query_split_leakage.
   • Dynamic features joined at training time: dynamic_feature_leakage; log at click time.

10. [interview] Is there any exploration mechanism? Branches:

    • None, and the system has retrained on its own clicks for multiple rounds: assume presentation_bias_echo_chamber is accumulating; the question is how much rather than whether.
    • Explore path present: ask how observations flow back to the exploit model and how the exploration budget is set.

11. [code, if cross-encoder rerank present] Does relevance in this domain depend on non-semantic signals (recency, popularity, price)? Branches:

    • Yes, cross-encoder is the final scorer: see retrieval#crossencoder_recency_blind; the intervention here is an LTR layer taking the cross-encoder score as one feature, or post-hoc reweighting.
    • Domain mismatch suspected: see retrieval#crossencoder_training_mismatch.

Interventions by likely cause

Ordered by frequency of cause. Cross-references to symptom IDs are in stage#symptom_id format.

Hypothesis	Supporting evidence	Intervention	Expected impact	Cost / risk
Labels were never validated as labels (upstream of every judgment symptom)	Click-derived judgments feed training directly; no label-level A/B ever ran	Deploy the judgments as a signals override and A/B test that before (re)training anything	Catches wrong labels on a cheap rollback; validates the entire pipeline’s foundation	Low: reuses existing boost machinery; the stage’s cheapest high-value move
Judgments built from raw CTR (ranking_and_fusion#position_bias_labels)	Grades correlate with legacy rank; clicks divided by raw impressions	Replace impressions with SDBN examines (at or above last click); drop or backfill zero-click sessions	Judgments start reflecting merit rather than inherited position	Low: an aggregation change, no model change
Raw ratios with no confidence (ranking_and_fusion#sparse_judgment_overconfidence)	Grades of 1.0 on tiny denominators; tail documents jump around between rounds	Beta prior on grades; posterior variance as ~1/variance sample weights in training	Confident and sparse evidence coexist in one training set at appropriate strength	Low: prior constants need tuning to signal volume; no procedure exists, start below a 0.5 default grade
Signals aggregation missing refinements (ranking_and_fusion#fragmented_signals, #signal_spam, #flat_signal_weights, #stale_signals)	Aggregation lacks normalization, user dedup, type weights, or decay	Add the missing refinement(s): normalize query text; one signal per user per (query, doc); weight click/cart/purchase ~1/10/25 with negatives; half-life decay matched to the domain	Each refinement eliminates its failure class; together they make head-query boosting trustworthy	Low: aggregation-job changes; weights and half-life need domain review
Boosting judged outside its scope (ranking_and_fusion#signals_long_tail_no_op)	“Doesn’t work” verdict from whole-distribution measurement, or transient corpus	Rescope measurement to head queries; LTR for the tail; cluster transient items into stable IDs where possible	Correct verdict on a working lever; tail handled by the right tool	Low for measurement; medium for LTR (its own subsystem)
Click data enters the score twice (ranking_and_fusion#signals_features_collision)	Signals feed both a boost and a model feature; rankings oscillate	Pick a composition rule: signals carry head queries, model carries the tail (route LTR to no-coverage queries), or accept the loop explicitly	Stable, debuggable composition	Low: routing logic; the cost is deciding rather than building
Train/test split leaks queries (ranking_and_fusion#query_split_leakage)	Strong offline metrics evaporate online	Split at the query level, never the document level	Offline evaluation becomes predictive again	Low: a pipeline fix, then re-baseline every prior offline result
Dynamic features captured at training time (ranking_and_fusion#dynamic_feature_leakage)	Offline-online gap concentrated in volatile-feature queries	Log feature values at click time (named-queries scores, pseudofields, computed fields); train on logged values	Removes a silent offline-online divergence	Medium: logging infrastructure plus judgment-pipeline change
Model rediscovers popularity (ranking_and_fusion#popularity_feature_loop)	Feature importance dominated by popularity; similarity negligible	Feature ablation to confirm; then a composition rule (accept the loop for head, exclude popularity features for tail routing), with exploration as the full fix	Model regains discriminative power on the tail	Medium: retraining rounds plus routing
Rerank windows unvalidated (ranking_and_fusion#rerank_depth_mistuned)	Depths copied from examples; relevant candidates land below the window, or the expensive phase dominates latency	Measure the pre-rerank rank distribution of relevant candidates; set each phase’s window to cover it; enforce cheapest-first ordering	Removes an invisible quality ceiling or reclaims wasted latency budget	Low: measurement plus constants; deeper windows cost linear rerank compute
Ranker locked in its own echo chamber (ranking_and_fusion#presentation_bias_echo_chamber)	Result classes narrow over rounds; new candidate shapes never gain ground; offline metrics fine	Add an explore path: Gaussian process over (features, grade, variance), acquisition function scoring, inject winners into live traffic (fixed slot or interleaved); adopt features into the exploit model once covered	Dark feature regions gain evidence; the training loop stops feeding on itself	High: live-traffic quality spent on exploration; a second modeling path to operate
Cross-encoder blind to non-semantic relevance (retrieval#crossencoder_recency_blind, retrieval#crossencoder_training_mismatch)	Stale results where recency matters; off-the-shelf reranker underperforms baseline on the target corpus	LTR layer combining the cross-encoder score with recency, popularity, price features; or time-decay reweighting after cross-encoder scoring; domain-matched or finetuned cross-encoder for the mismatch case	Non-semantic relevance signals reach the final ordering	Medium: LTR is its own subsystem; reweighting is the cheap interim step