# Ranking and Fusion
Search Engineering · Diagnostic Runbook · Chapter 5
Interactive edition: https://mutapa.dev/search-engineering/reference/ranking-and-fusion
Find where labels, behavioral signals, and reranking stages distort the final order.
## Start with the common causes
1. [The ranker trains on its own output](https://mutapa.dev/search-engineering/reference/ranking-and-fusion.md#presentation_bias_echo_chamber)
Already-visible result types keep gaining evidence and position.
2. [Clicks are treated as unbiased judgments](https://mutapa.dev/search-engineering/reference/ranking-and-fusion.md#position_bias_labels)
Training labels mostly reproduce the previous ranking order.
3. [The expensive stage sees the wrong window](https://mutapa.dev/search-engineering/reference/ranking-and-fusion.md#rerank_depth_mistuned)
Relevant candidates sit just outside the reranking depth.
## Browse all symptoms
### Biased training evidence
- [The ranker trains on its own output](https://mutapa.dev/search-engineering/reference/ranking-and-fusion.md#presentation_bias_echo_chamber): Click-trained ranking keeps promoting the same class of results; new or differently-shaped candidates never gain ground; offline metrics look fine because they are computed on the same biased labels.
- [Judgments inherit the legacy ranker's ordering](https://mutapa.dev/search-engineering/reference/ranking-and-fusion.md#position_bias_labels): Click-derived judgments strongly correlate with the legacy ranker's ordering. Items ranked high by the old system look relevant regardless of merit; CTR was computed over raw impressions.
- [The model rediscovers its own label source](https://mutapa.dev/search-engineering/reference/ranking-and-fusion.md#popularity_feature_loop): Feature importance shows popularity dominant and semantic similarity negligible. The model has rediscovered the proxy that generated its own labels.
### Training and validation
- [Raw ratios carry no confidence](https://mutapa.dev/search-engineering/reference/ranking-and-fusion.md#sparse_judgment_overconfidence): Rarely-shown documents with one or two lucky clicks outrank consistently-validated ones. Judgment grades of 1.0 sit on a denominator of 1; no prior, no confidence weighting.
- [Features computed at training time rather than click time](https://mutapa.dev/search-engineering/reference/ranking-and-fusion.md#dynamic_feature_leakage): Model performs well offline, poorly online; the gap concentrates in queries where volatile features (price, availability, badges) changed between click time and training time.
- [Train/test split at the wrong level](https://mutapa.dev/search-engineering/reference/ranking-and-fusion.md#query_split_leakage): Suspiciously strong offline metrics that evaporate in A/B tests. Train/test split was done at the document level; query-dependent feature vectors leak across the split.
### Signal quality
- [One user moves a head query](https://mutapa.dev/search-engineering/reference/ranking-and-fusion.md#signal_spam): A product or document climbs a head query's results with signal traceable to one or a few users. No per-user deduplication in the signals aggregation.
- [Query variants split their evidence](https://mutapa.dev/search-engineering/reference/ranking-and-fusion.md#fragmented_signals): Boost tables full of near-duplicate keys ('iPad', 'ipad', 'iPad 2'), each with weak counts; boosting underperforms on queries that are obviously popular. No query normalization in the GROUP BY.
- [Yesterday's winners never age out](https://mutapa.dev/search-engineering/reference/ranking-and-fusion.md#stale_signals): Last season's winners still rank on time-dated queries; users see the old holiday schedule. No half-life decay, or a half-life wrong by orders of magnitude for the domain.
- [All signals count the same](https://mutapa.dev/search-engineering/reference/ranking-and-fusion.md#flat_signal_weights): Boost values dominated by abundant low-confidence clicks; purchases barely move rankings; negative signals absent entirely. All signal types counted equally.
### Pipeline composition
- [Boosting judged outside its scope](https://mutapa.dev/search-engineering/reference/ranking-and-fusion.md#signals_long_tail_no_op): Team reports 'signals boosting doesn't work'; measurement was on the whole query distribution, or the corpus is transient so per-document signal never accumulates. A structural scope limit rather than a tuning failure.
- [Click data enters the score twice](https://mutapa.dev/search-engineering/reference/ranking-and-fusion.md#signals_features_collision): Signals boost and LTR model disagree; rankings oscillate or the model appears to fight manual boosts. The same click-derived information enters the final score twice.
- [Phase windows chosen by copy-paste](https://mutapa.dev/search-engineering/reference/ranking-and-fusion.md#rerank_depth_mistuned): Relevant candidates sit just below the rerank window and never surface, or latency blows up because an expensive scorer runs over a set a cheaper phase should have narrowed first.
## Deeper inspection and interventions
## Concept
The ranking and fusion stage takes the candidate set produced by retrieval and reorders it into the list the user (or the generation stage) actually sees. Retrieval optimizes for recall at tolerable cost; this stage optimizes for precision at the top of the list, using scorers that are too expensive or too data-hungry to run over the whole corpus. It is the last stage that reorders: everything before it widens or shapes the candidate pool, everything after it consumes the ordering.
The stage's organizing frame is the *cost-ordered phase pipeline*. Ranking is a sequence of phases, each rescoring a strictly smaller candidate set, ordered cheapest first: a cheap candidate phase (BM25, quantized ANN) narrows the corpus to hundreds or thousands; a middle phase (learning-to-rank, signals boosts, filters) narrows to the low hundreds; the most expensive phase (cross-encoder, full-precision vector rescoring, or an LLM) touches tens. The rule is simple: never run an expensive scorer on a candidate set a cheaper scorer could have narrowed first. Cost means compute, latency, and memory together; they usually move as one. A corollary that matters diagnostically: each phase's depth caps what later phases can fix. A cross-encoder reranking the top 10 cannot rescue a relevant document at rank 40.
Write a candidate budget beside every phase. Counts such as 1,000 retrieved, 100 feature-scored, 20 cross-encoder reranked, and 5 shown are useful starting hypotheses rather than portable defaults. On a labeled set, record where relevant documents land before each phase and set the windows to cover that distribution. A later model cannot rescue a candidate an earlier window excluded.
The same compatibility rule applies to ranking features. Construct historical training rows from the values available when the impression happened rather than from today's price, stock, popularity, or user profile. Training on current values leaks information the serving ranker could not have known. The safest path is one versioned transformation contract shared by feature logging, offline training, and online scoring.
Three scorer families fill those phases, and the source course prescribes an explicit order for adopting them: *memorize before you generalize*. Signals boosting memorizes right answers for head queries by aggregating user behavior per (query, document) pair into an additive boost. Learning to rank generalizes to the tail by training a model (LambdaMART on gradient-boosted trees is the modern classic) over a small hand-engineered feature set, using judgments typically derived from those same clicks. Cross-encoder reranking supplies the semantic final phase where query-document similarity dominates relevance. Getting signals boosting solid on head queries first, verifying it in A/B tests, then training LTR for the tail is the prescribed sequence; skipping straight to LTR is the named anti-pattern. The stage also carries a second sense of fusion beyond the retriever-output fusion documented at Stage 3: composing heterogeneous score sources (base relevance, boost, model score) into one final ordering without letting the same evidence count twice.
**What good looks like at this stage:** the top of the list is right for the queries users actually run, at a latency the product can afford, and the training loop that produces the ranking does not simply reflect the previous ranking back at itself.
## Symptom catalog
The catalog leads with judgment-construction failures. They silently corrupt everything downstream, and they are the least likely to be suspected because the symptoms surface as "the model isn't good enough" rather than "the labels are wrong."
### `presentation_bias_echo_chamber`: The ranker trains on its own output
- **Observable signal:** Click-trained ranking keeps promoting the same class of results; new or differently-shaped candidates never gain ground; offline metrics look fine because they are computed on the same biased labels.
- **Frequency:** Universal in click-trained systems without an exploration mechanism. A matter of degree rather than presence.
- **Cost of leaving unaddressed:** Whole regions of feature space stay dark forever. If the search is bad, click-trained ranking promotes more of the same badness, with offline metrics reporting success throughout.
Training data only contains what the current ranker showed. If the ranker never surfaces a class of documents (say, high overview-field BM25 with low title-field BM25), users cannot click them, so no evidence for that region ever enters the judgments. Position-bias corrections do not reach this; they debias clicks among shown results, and this failure is about results never shown. The escape is exploration: fit a Gaussian process on (features, grade, variance) triples so unobserved feature regions get a predicted grade and a predicted uncertainty, score candidates with an acquisition function (high predicted grade with high variance is the sweet spot), and inject winners into live traffic to harvest clicks the production ranker would never have generated. The mature form is a two-path architecture: an exploit path (production LTR on a minimal proven feature set) and an explore path (a Gaussian process over a feature superset) feeding observations back. Full treatment of the judgment-construction side lives at `evaluation_and_feedback#presentation_bias_echo_chamber`.
### `position_bias_labels`: Judgments inherit the legacy ranker's ordering
- **Observable signal:** Click-derived judgments strongly correlate with the old system's ranking. Items the legacy ranker put on top look relevant regardless of merit; the "best" documents for a query are suspiciously often the ones that were already ranked first.
- **Frequency:** Very common. Raw CTR (clicks over impressions) is the naive baseline nearly every team starts from.
- **Cost of leaving unaddressed:** The new model learns to imitate the old ranker. Training spend produces a ranker that reproduces the system it was meant to replace.
Users click what they see, and they see the top. Raw clicks over raw impressions rewards inherited rank. The fix is the examines correction: replace impressions with examines, defined as everything at or above the last click in a session. Items below the last click are treated as unseen and leave the denominator, so a result clicked at rank 19 earns a strong ratio precisely because it was rarely examined. This is the Simplified Dynamic Bayesian Network (SDBN), which captures roughly 90% of the full DBN's accuracy at a fraction of the complexity. Two boundary conditions to check while inspecting: the session definition (one query and the actions on that result page; a query change starts a new session) and zero-click sessions, which are invisible to SDBN and must be dropped or backfilled with softer pre-click events (hover, dwell, scroll). Full treatment at `evaluation_and_feedback#position_bias_unaccounted`, the same defect viewed from the judgment pipeline.
### `sparse_judgment_overconfidence`: Raw ratios carry no confidence
- **Observable signal:** Rarely-shown documents with one or two lucky clicks outrank consistently-validated ones. The judgment table contains grades of 1.0 sitting on a denominator of 1.
- **Frequency:** Very common wherever judgments come from clicks and the query distribution has any tail.
- **Cost of leaving unaddressed:** The training set treats a single lucky click and a hundred consistent clicks as identical evidence. Model quality degrades exactly where data is thinnest.
One click on one examine and 100 clicks on 100 examines both divide to 1.0 with wildly different confidence. The fix is a Beta prior: seed each (query, document) with pseudo-counts (the lesson's illustration is alpha 15, beta 20, a deliberate below-0.5 default, innocent until proven relevant), increment alpha with clicked-examines and beta with unclicked-examines. The posterior mean becomes the grade; the posterior variance becomes the confidence, entering LambdaMART or XGBoost as sample weights of roughly 1/variance. The lessons give no procedure for choosing the prior constants; tune them to signal volume and treat the published numbers as an illustration. Full treatment at `evaluation_and_feedback#sparse_judgment_overconfidence`.
### `popularity_feature_loop`: The model rediscovers its own label source
- **Observable signal:** Feature importance shows popularity dominant and semantic similarity negligible. Rankings drift toward pure popularity ordering over successive retraining rounds.
- **Frequency:** Common in any LTR system that both derives labels from clicks and feeds click-derived popularity in as a feature.
- **Cost of leaving unaddressed:** A self-reinforcing loop: popularity drives clicks, clicks drive labels, labels train popularity importance. The tail gets ranked by a signal that has no information there.
This is a within-model cousin of the echo chamber. Some teams accept the loop deliberately for head queries: strong signals carry the head, and the model's non-popularity features carry the tail. If the loop is unintentional, the diagnostic move is feature ablation (train on random feature subsets and compare) followed by a deliberate composition rule; the full mitigation is the exploration machinery under `presentation_bias_echo_chamber`.
### `dynamic_feature_leakage`: Features computed at training time rather than click time
- **Observable signal:** Model performs well offline, poorly online, and the gap concentrates in queries where volatile features (price, availability, promotion badges) changed between the click and the training run.
- **Frequency:** Common wherever features are joined onto judgments from a live table at training time.
- **Cost of leaving unaddressed:** Labels are attached to a state of the world that no longer exists. The model learns relationships that were never true at decision time.
The fix is feature logging discipline: for dynamic features, log the feature value at click time and train on the logged value. Engines support returning per-feature scores in the search request itself (OpenSearch/Elasticsearch named queries with `include_named_queries_score`, Solr pseudofields, Vespa computed fields), which makes click-time logging a one-request pattern rather than a separate pipeline.
### `query_split_leakage`: Train/test split at the wrong level
- **Observable signal:** Suspiciously strong offline metrics that evaporate in A/B tests.
- **Frequency:** Common in first LTR implementations; the default `train_test_split` over rows produces it silently.
- **Cost of leaving unaddressed:** Offline evaluation is meaningless. Every modeling decision made against it is unvalidated.
Feature vectors in LTR are query-dependent: the same document has different features under different queries, and rows from one query are strongly correlated. A document-level (row-level) split puts rows from the same query on both sides, and the model is graded partly on queries it saw in training. Split at the query level, always.
### `signal_spam`: One user moves a head query
- **Observable signal:** A product or document climbs a head query's results, and the signal behind the climb traces to one or a few user IDs.
- **Frequency:** Common in any signals-boosted system without per-user deduplication; guaranteed eventually in adversarial domains (marketplaces, anywhere sellers benefit from rank).
- **Cost of leaving unaddressed:** Head queries, the exact queries signals boosting exists to protect, become manipulable at the cost of one obsessive or adversarial user.
The fix is layered rather than absolute. First, deduplicate per user: add user_id to the aggregation's GROUP BY and count one signal per user per (query, document), which raises the attack cost from one account clicking repeatedly to thousands of fake accounts. Second, weight purchases and other paid actions above clicks, which raises the attack cost to real money.
### `fragmented_signals`: Query variants split their evidence
- **Observable signal:** Boost tables full of near-duplicate keys ("iPad", "ipad", "Ipad", "iPad 2"), each with weak counts. Boosting underperforms on queries that are obviously popular.
- **Frequency:** Very common; it is the first defect in any naive aggregation.
- **Cost of leaving unaddressed:** Signal that should pool into a strong boost fragments into many weak ones, none of which clears the noise floor. The head of the distribution behaves like the tail.
The fix is query normalization inside the aggregation: lowercase, collapse whitespace, optionally stem, before grouping. This is the first of lesson 3's four refinements to the naive GROUP BY, and the cheapest.
### `stale_signals`: Yesterday's winners never age out
- **Observable signal:** Last year's or last season's winners still rank on time-dated queries. Users searching for the holiday schedule get the old holiday schedule.
- **Frequency:** Common; decay is the refinement teams most often skip because the failure takes months to appear.
- **Cost of leaving unaddressed:** Boosts calcify. The system's memory of past relevance overrides present relevance exactly on the queries where users notice.
The fix is half-life decay in the aggregation: `weight * 0.5^(age / half_life)`. The half-life is domain-specific by orders of magnitude (hours for news, roughly a month for job boards, weeks to months for e-commerce), so a copied constant can be as harmful as no decay. Done right, decay plus fresh clicks makes time-dated head queries self-correct: the new year's document overtakes the old one with zero engineering intervention.
### `flat_signal_weights`: All signals count the same
- **Observable signal:** Boost values dominated by abundant low-confidence clicks; purchases and other strong signals barely move rankings; negative signals (skips, cart removals, returns) are absent entirely.
- **Frequency:** Common in first implementations that count events rather than weighting them.
- **Cost of leaving unaddressed:** The boost measures attention rather than satisfaction. Clickbait-shaped results win over results people actually wanted.
Weight signal types by confidence in what they say about relevance. The canonical starting ratios from lesson 3: click 1, add-to-cart 10, purchase 25, with negative weights for skips, removals, and returns. Weights are set by domain intuition plus qualitative review more than by exhaustive search. Note the boundary with judgment construction: weighted sums are fine for boost tables, but when building training labels, deeper-funnel events work better as confirmations or guardrails on the click than as weighted addends, because the further an event sits from the search action, the more confounders it picks up.
### `signals_long_tail_no_op`: Boosting judged outside its scope
- **Observable signal:** Team reports "signals boosting doesn't work." On inspection, the measurement was over the whole query distribution, or the corpus is transient (marketplace stock-of-one listings, job postings, news) so per-document signal never accumulates.
- **Frequency:** Common as a conclusion; the underlying scope limit is universal.
- **Cost of leaving unaddressed:** A working technique gets ripped out, or worse, effort goes into tuning a lever that is structurally incapable of reaching the problem.
Signals boosting works only at the head of the query distribution and only for stable documents. Long-tail queries never accumulate enough signal; transient corpora never let signal accumulate per document. This is a structural limit rather than a tuning failure. The interventions: measure boosting on head queries specifically; for the tail, LTR is the tool that generalizes past the limit; for transient corpora, cluster short-lived items into stable abstract IDs where possible so signal can accumulate at the cluster level.
### `signals_features_collision`: Click data enters the score twice
- **Observable signal:** Signals boost and LTR model disagree; rankings oscillate between retraining rounds, or the model appears to fight manual boosts.
- **Frequency:** Common once a system has both a boost layer and a learned model, which is exactly the state the memorize-then-generalize path leads to.
- **Cost of leaving unaddressed:** Double-counted evidence and unstable composition. Debugging any single ranking becomes guesswork because two components encode the same facts with different lags.
When click-derived signals feed both an additive boost and an LTR feature, the model and the boost encode the same CTR-derived information and can fight each other. Pick a composition rule deliberately. The pragmatic pattern: let strong signals carry head queries directly and let the model's non-popularity features carry the tail, either by feeding signals in as a feature and accepting the loop knowingly, or by routing LTR only to queries without signal coverage.
### `rerank_depth_mistuned`: Phase windows chosen by copy-paste
- **Observable signal:** Either relevant candidates sit just below the rerank window and never surface (too shallow), or latency budgets blow up on the expensive phase (too deep, or the cost ordering is inverted and an expensive scorer runs over a set a cheaper phase should have narrowed).
- **Frequency:** Common; depths are usually copied from a tutorial or a paper and never revisited.
- **Cost of leaving unaddressed:** Shallow windows put a hard ceiling on ranking quality that no model improvement can lift; deep windows spend the latency budget where it buys nothing.
The lessons supply worked depths as starting points: roughly the top 1,000 to 1,500 candidates pulled into an external LTR service for the middle phase, the top 50 to 100 fed to a cross-encoder for the expensive phase, and 2x to 10x over-fetch for quantized retrieval feeding full-precision rescoring (mechanics at Stage 3, `retrieval#underfetch_quantized`). The diagnostic that replaces copy-paste: measure where relevant candidates actually land in the pre-rerank ordering, and set each window to cover that distribution. If relevant documents routinely land at rank 150 out of the candidate phase, a top-100 cross-encoder window loses them by construction.
### Cross-referenced symptoms (owned by Stage 3)
Two cross-encoder failure modes present at this stage but are anchored in the retrieval chapter, where the retrieval-plus-rerank path is inspected:
- **`retrieval#crossencoder_training_mismatch`**: an off-the-shelf cross-encoder trained on general web QA reranks the target corpus worse than the baseline.
- **`retrieval#crossencoder_recency_blind`**: cross-encoders score semantic relevance only; newer equivalents rank below older content wherever recency carries relevance.
Stage 4 carries the intervention side of both: a learning-to-rank layer that takes the cross-encoder score as one feature among several (recency, popularity, price), or a post-hoc reweighting applied after cross-encoder scoring. Tree-based LTR wins where ranking depends on statistics; cross-encoders win where ranking is dominated by semantic similarity; the standard bridge is feeding embedding cosine similarity into LTR as one feature.
## How to inspect
### By code pattern
Greppable signals that reveal ranking-stage configuration choices.
| Pattern / location | What you're looking for | What suggests a problem |
|---|---|---|
| Signals aggregation job (SQL `GROUP BY` over clickstream, or equivalent) | The four refinements: query normalization, user dedup, type weights, decay | Any refinement missing maps to a symptom: no `lower()`/`trim()` suggests `fragmented_signals`; no `user_id` in the GROUP BY suggests `signal_spam`; no per-type weights suggests `flat_signal_weights`; no decay term suggests `stale_signals` |
| `0.5 **`, `half_life`, `decay` in aggregation code | The half-life constant | Absent entirely; or a constant copied across domains (a news system with a 30-day half-life, or vice versa) |
| `clicks / impressions`, `ctr` in judgment construction | The position-bias treatment | Raw CTR with no examines logic suggests `position_bias_labels`; look for last-click session truncation to confirm SDBN is present |
| `alpha`, `beta`, prior constants near grade computation; `sample_weight` in the training call | Sparsity handling | Grades computed as raw ratios with no prior suggests `sparse_judgment_overconfidence`; priors present but sample weights unused wastes the variance information |
| `train_test_split` over judgment rows | Split level | Row-level or document-level split suggests `query_split_leakage`; correct code groups by query ID first |
| Feature joins in the training pipeline (`JOIN ... ON current_`, live price/stock lookups) | When dynamic features are captured | Features joined from live tables at training time suggests `dynamic_feature_leakage`; healthy pipelines log feature values at click time |
| `include_named_queries_score`, Solr pseudofields, Vespa computed fields | Feature logging mechanism | Absent in a system with dynamic features; features recomputed offline instead |
| `rank:pairwise`, `lambdarank`, `xgboost`, `lightgbm` objective config | LTR model class | Pointwise regression objective in production (pairwise almost always outperforms pointwise for ranking) |
| Rerank window constants (`rescore`, `window_size`, `top_n` into LTR service or cross-encoder) | Phase depths | Values matching a tutorial exactly, never validated against where relevant candidates land pre-rerank; suggests `rerank_depth_mistuned` |
| Feature logging and training joins | Point-in-time construction of volatile features | Live price, inventory, popularity, or user features joined at training time rather than the values available when the impression occurred |
| Order of scoring phases in the query path | Cost ordering | An expensive scorer (cross-encoder, LLM) running before or instead of a cheap narrowing phase |
| Signals appearing both in a boost clause and in the LTR feature list | Score composition | Click-derived data entering the final score twice suggests `signals_features_collision` |
| Boost application point: sidecar lookup at query time vs boost fields in the index mapping | Query-time vs index-time boosting | Either is valid; query-time plus deep paging is a known pain point, index-time means every aggregation triggers re-indexing |
| Feature list size and contents | Classic LTR feature engineering | Dozens-plus features without ablation; popularity-type features present alongside click-derived labels suggests `popularity_feature_loop` |
### By measurement
Metrics, logs, and traces to enable for ranking diagnosis.
| Measurement | How to compute | Healthy range / red-flag threshold |
|---|---|---|
| Judgment-to-legacy-rank correlation | Correlate click-derived grades with the position the legacy ranker served each (query, doc) at | Strong positive correlation suggests `position_bias_labels`; after SDBN it should weaken visibly |
| Judgment denominator distribution | Histogram of examine counts behind each grade | A large mass of grades on denominators below ~10 with no prior or sample weighting suggests `sparse_judgment_overconfidence` |
| Offline nDCG vs online A/B delta | Compare offline eval gain against the shipped A/B result, per model version | Repeated large offline-online gaps suggest `query_split_leakage` or `dynamic_feature_leakage`; localize by checking whether the gap concentrates in volatile-feature queries |
| Feature importance report | Standard gain/split importance from the trained trees, per retraining round | Popularity dominant and similarity negligible suggests `popularity_feature_loop`; track drift across rounds |
| Boost key cardinality and near-duplicate rate | Count boost-table keys; cluster case/whitespace variants | High near-duplicate rate suggests `fragmented_signals` |
| Per-user signal concentration | For each boosted (query, doc), max share of signal from one user | Any (query, doc) where one user contributes a dominant share suggests `signal_spam` |
| Signal age distribution | Histogram of signal event ages weighted by their current boost contribution | Old events still carrying large weight on time-dated queries suggests `stale_signals` |
| Boost lift, head vs tail | Measure boosting's ranking lift separately on head queries and tail queries | Boosting judged on the whole distribution understates it; near-zero lift on the head is the real red flag, near-zero on the tail is expected (`signals_long_tail_no_op`) |
| Pre-rerank rank of relevant candidates | For a labeled query set, record where relevant docs land in the candidate ordering before each rerank phase | Relevant candidates routinely landing below a phase's window suggests `rerank_depth_mistuned` |
| Result-class diversity over retraining rounds | Track the distribution of served results across feature-space regions (or simpler proxies: source field, document class) per round | Monotonic narrowing suggests `presentation_bias_echo_chamber` |
| Per-phase latency | p50/p95 per scoring phase in traces | The expensive phase dominating while operating on an un-narrowed set indicates inverted cost ordering |
## Diagnostic questions
Triage flow for an agent or engineer assessing this stage. Label provenance comes first: no model question is interpretable until the origin of the grades is established.
1. **[interview] Where do the ranking judgments come from?** Human-labeled, click-derived, or none (no learned ranking). Branches:
- Click-derived: continue to questions 2 through 4; they are the highest-yield checks in the stage.
- Human-labeled: ask about coverage and refresh cadence, then skip to question 5.
- No learned ranking: skip to questions 5 and 8; this stage reduces to signals hygiene and phase depths.
2. **[interview] Were the labels themselves ever A/B tested?** The training set is itself a model. Deploying the click-derived judgments directly as a signals override (pin the top-graded results for their queries) and A/B testing that is the stage's cheapest high-value move. Branches:
- Never: recommend it before any further model work; if the labels are wrong, this catches it on a cheap rollback instead of inside a trained model.
- Tested and won: label quality has evidence; move on.
3. **[code] How is position bias handled in judgment construction?** Branches:
- Raw clicks over raw impressions: `position_bias_labels`; introduce SDBN examines.
- SDBN present: check the session definition (query change starts a new session) and how zero-click sessions are handled.
4. **[code] How are sparse judgments handled?** Branches:
- Raw ratios, no prior: `sparse_judgment_overconfidence`; introduce a Beta prior and variance-derived sample weights.
- Prior present: ask whether the variance feeds sample weights or is computed and discarded.
5. **[code] What does the final score compose, and does click-derived data enter more than once?** Branches:
- Boost and model both consume clicks: `signals_features_collision`; establish a composition rule (signals carry head, model carries tail).
- Single entry point: confirm the composition is applied in cost order.
6. **[code] In the signals aggregation, which of the four refinements are present: query normalization, per-user dedup, type weights, decay?** Each absence maps to a symptom (`fragmented_signals`, `signal_spam`, `flat_signal_weights`, `stale_signals`). If decay is present, ask how the half-life was chosen; a copied constant is a soft red flag.
7. **[interview] Where is signals boosting being measured?** Branches:
- Whole query distribution, or a transient corpus: `signals_long_tail_no_op`; rescope the measurement to head queries, use LTR for the tail.
- Head queries specifically: the measurement is sound.
8. **[code] What are the phase depths, and where do relevant candidates land pre-rerank?** Branches:
- Depths copied, landing distribution unmeasured: `rerank_depth_mistuned`; measure before tuning.
- Expensive scorer running over an un-narrowed set: cost-ordering violation; insert or deepen a cheaper phase first.
9. **[code] For the LTR pipeline: split level, and feature capture time?** Branches:
- Document-level split: `query_split_leakage`.
- Dynamic features joined at training time: `dynamic_feature_leakage`; log at click time.
10. **[interview] Is there any exploration mechanism?** Branches:
- None, and the system has retrained on its own clicks for multiple rounds: assume `presentation_bias_echo_chamber` is accumulating; the question is how much rather than whether.
- Explore path present: ask how observations flow back to the exploit model and how the exploration budget is set.
11. **[code, if cross-encoder rerank present] Does relevance in this domain depend on non-semantic signals (recency, popularity, price)?** Branches:
- Yes, cross-encoder is the final scorer: see `retrieval#crossencoder_recency_blind`; the intervention here is an LTR layer taking the cross-encoder score as one feature, or post-hoc reweighting.
- Domain mismatch suspected: see `retrieval#crossencoder_training_mismatch`.
## Interventions by likely cause
Ordered by frequency of cause. Cross-references to symptom IDs are in `stage#symptom_id` format.
| Hypothesis | Supporting evidence | Intervention | Expected impact | Cost / risk |
|---|---|---|---|---|
| **Labels were never validated as labels** (upstream of every judgment symptom) | Click-derived judgments feed training directly; no label-level A/B ever ran | Deploy the judgments as a signals override and A/B test that before (re)training anything | Catches wrong labels on a cheap rollback; validates the entire pipeline's foundation | Low: reuses existing boost machinery; the stage's cheapest high-value move |
| **Judgments built from raw CTR** (`ranking_and_fusion#position_bias_labels`) | Grades correlate with legacy rank; clicks divided by raw impressions | Replace impressions with SDBN examines (at or above last click); drop or backfill zero-click sessions | Judgments start reflecting merit rather than inherited position | Low: an aggregation change, no model change |
| **Raw ratios with no confidence** (`ranking_and_fusion#sparse_judgment_overconfidence`) | Grades of 1.0 on tiny denominators; tail documents jump around between rounds | Beta prior on grades; posterior variance as ~1/variance sample weights in training | Confident and sparse evidence coexist in one training set at appropriate strength | Low: prior constants need tuning to signal volume; no procedure exists, start below a 0.5 default grade |
| **Signals aggregation missing refinements** (`ranking_and_fusion#fragmented_signals`, `#signal_spam`, `#flat_signal_weights`, `#stale_signals`) | Aggregation lacks normalization, user dedup, type weights, or decay | Add the missing refinement(s): normalize query text; one signal per user per (query, doc); weight click/cart/purchase ~1/10/25 with negatives; half-life decay matched to the domain | Each refinement eliminates its failure class; together they make head-query boosting trustworthy | Low: aggregation-job changes; weights and half-life need domain review |
| **Boosting judged outside its scope** (`ranking_and_fusion#signals_long_tail_no_op`) | "Doesn't work" verdict from whole-distribution measurement, or transient corpus | Rescope measurement to head queries; LTR for the tail; cluster transient items into stable IDs where possible | Correct verdict on a working lever; tail handled by the right tool | Low for measurement; medium for LTR (its own subsystem) |
| **Click data enters the score twice** (`ranking_and_fusion#signals_features_collision`) | Signals feed both a boost and a model feature; rankings oscillate | Pick a composition rule: signals carry head queries, model carries the tail (route LTR to no-coverage queries), or accept the loop explicitly | Stable, debuggable composition | Low: routing logic; the cost is deciding rather than building |
| **Train/test split leaks queries** (`ranking_and_fusion#query_split_leakage`) | Strong offline metrics evaporate online | Split at the query level, never the document level | Offline evaluation becomes predictive again | Low: a pipeline fix, then re-baseline every prior offline result |
| **Dynamic features captured at training time** (`ranking_and_fusion#dynamic_feature_leakage`) | Offline-online gap concentrated in volatile-feature queries | Log feature values at click time (named-queries scores, pseudofields, computed fields); train on logged values | Removes a silent offline-online divergence | Medium: logging infrastructure plus judgment-pipeline change |
| **Model rediscovers popularity** (`ranking_and_fusion#popularity_feature_loop`) | Feature importance dominated by popularity; similarity negligible | Feature ablation to confirm; then a composition rule (accept the loop for head, exclude popularity features for tail routing), with exploration as the full fix | Model regains discriminative power on the tail | Medium: retraining rounds plus routing |
| **Rerank windows unvalidated** (`ranking_and_fusion#rerank_depth_mistuned`) | Depths copied from examples; relevant candidates land below the window, or the expensive phase dominates latency | Measure the pre-rerank rank distribution of relevant candidates; set each phase's window to cover it; enforce cheapest-first ordering | Removes an invisible quality ceiling or reclaims wasted latency budget | Low: measurement plus constants; deeper windows cost linear rerank compute |
| **Ranker locked in its own echo chamber** (`ranking_and_fusion#presentation_bias_echo_chamber`) | Result classes narrow over rounds; new candidate shapes never gain ground; offline metrics fine | Add an explore path: Gaussian process over (features, grade, variance), acquisition function scoring, inject winners into live traffic (fixed slot or interleaved); adopt features into the exploit model once covered | Dark feature regions gain evidence; the training loop stops feeding on itself | High: live-traffic quality spent on exploration; a second modeling path to operate |
| **Cross-encoder blind to non-semantic relevance** (`retrieval#crossencoder_recency_blind`, `retrieval#crossencoder_training_mismatch`) | Stale results where recency matters; off-the-shelf reranker underperforms baseline on the target corpus | LTR layer combining the cross-encoder score with recency, popularity, price features; or time-decay reweighting after cross-encoder scoring; domain-matched or finetuned cross-encoder for the mismatch case | Non-semantic relevance signals reach the final ordering | Medium: LTR is its own subsystem; reweighting is the cheap interim step |