Chapter 4 ended with a fused candidate set for “a space movie where the dad ages slowly”. Dense retrieval recovered Interstellar, and fusion kept it among the leading candidates, which gives the next stage a chance to place it first. That chance can still be wasted. The user sees an ordered list rather than the candidate set, so placing weaker films above Interstellar can hide the retrieval success from view. Ranking begins with that unfinished result: the right film is present, but its position has not been earned.
Earning that position requires evidence the retrievers did not consider. Their scores describe lexical or semantic fit, while behavior and domain features can show whether a film is popular, current, available to this viewer, or suited to a requested language or runtime. The ranker could make a better decision from all of this evidence if every signal cost the same to apply. It does not. Reading a stored popularity value is cheap, while a model that examines the query and film summary together must run again for every pair. A production pipeline therefore starts with inexpensive evidence across many candidates, then spends more computation on each smaller group that survives. Every cutoff saves time by making an exclusion permanent, so the chapter’s problem is to improve the order without removing Interstellar before the evidence needed to judge it can reach the decision.
Set the candidate budget
A cost-ordered pipeline becomes operational when every phase has a candidate window, the maximum number of documents it can pass to the next scorer. One request path might retrieve 1,000 candidates, compute richer features for 100, rerank 20 with a cross-encoder, and show five results or send five passages to a generator. These counts turn a sequence of models into an explicit allocation of computation.
Each window sets both a latency budget and a recall limit. A larger window gives the next phase more opportunities to recover a relevant document, but its cost usually grows with every query-document pair it must score. A smaller window saves time by making some mistakes permanent. If the relevant document enters at rank 150 and the next scorer receives only 100 candidates, changing that scorer cannot repair the result because the document never reaches it.
The film at rank 150 turns window sizing into a measurement problem. Across a set of labeled queries, the team can record where each relevant document appears before the cutoff, then compare how much coverage each candidate depth buys for its added latency. The resulting curve shows the price of giving the next scorer access to more relevant documents.
Coverage alone does not show whether the scorer can use that access. If relevant documents cluster just beyond the cutoff, widening the window or improving the preceding phase creates a real opportunity to repair the order. If the scorer rarely moves candidates near the bottom of its window, those extra inferences add cost without changing the final list. The window earns its depth only where survival leads to a measurable ranking improvement.
Recover evidence lost to compression
Candidate windows allocate model computation, but the widest retrieval window has another cost: the memory and bandwidth required to search stored vectors. Chapter 2 introduced quantization as an indexing decision. A 384-dimensional vector occupies 1,536 bytes in float32, 384 bytes in int8, or 48 bytes when reduced to one bit per dimension. Lower precision allows more vectors to remain in memory and makes the first comparison cheaper, but the compressed representation can change which documents appear nearest.
The instrument measures that change against the full-precision ranking. It selects 60 film vectors as queries, finds each one’s top 10 neighbors with float32 cosine similarity, then checks how many of those neighbors binary search recovers. The resulting recall@10 is an overlap measure between two retrieval methods. It does not report whether either list is relevant to a person. Direct binary search recovers an average of 44.8 percent of the float32 top ten. Select “Binary”, then choose “Over-fetch + rescore”, and watch the overlap measure change.
Select a vector precision and watch the storage footprint shrink, then compare its returned neighbors with the float32 top ten. For Binary, switch recovery strategies to see what a wider first pass and float32 rescoring can restore.
With “Over-fetch + rescore”, the first pass retrieves 40 candidates using the one-bit vectors instead of returning its top 10 directly. A second pass rescores those 40 candidates with the retained float32 vectors and returns the best 10. On this sample, overlap with the float32 baseline rises from 0.448 to 0.832. The second pass can repair the order only when the missing neighbor survived inside the wider binary window.
That recovery makes the budget visible. A wider first pass improves the chance that full-precision rescoring can restore the baseline neighbors, but it increases candidate transfer, memory access, and reranking latency. The appropriate depth comes from the recovery curve measured on representative queries under the system’s latency target. The full-precision vectors must also remain available for rescoring; discarding them at index time makes the compression loss permanent. Quantization establishes the pattern that the remaining stages will follow: use cheap evidence broadly, preserve the candidate set needed for recovery, and spend more only on the survivors.
Add behavioral and domain features
Retrieval scores explain how well a film summary matches the request, but a strong match can still lead to a poor order. For the
Signals boosting turns those events into a query-film feature. An offline aggregation groups events by a normalized query and film, assigns each event a weight, and stores the total for a cheap lookup during ranking. A starting scheme might assign an open a score of 1, a watchlist save 10, and a completed viewing 25, while a skip or quick return contributes a negative value. These ratios express assumptions about how strongly each action indicates success, so they must be checked against the product’s actual viewing journey rather than treated as universal constants.
The aggregation also needs controls for identity and time. Query normalization lets capitalization and punctuation variants of the w is its original weight, t is its age, and h is the chosen half-life. Interest around a new release may change within days, while viewing patterns for a classic can remain useful for months. The correct interval follows how quickly relevance changes for that query and catalog.
Aggregation makes this behavioral evidence cheap enough to apply across a broad candidate window, but only after a query-film pair has accumulated a useful history. A frequent request can gather enough interactions to strengthen Interstellar’s position, while a rare query or newly added film begins with little or no behavior to learn from and a changing catalog can make older observations misleading. Because missing behavior does not establish that a film is irrelevant, the same ranking phase needs features that apply before the first interaction and remain current as the catalog changes.
The corpus supplies some of those features, including release year, runtime, and original language, while a viewing service would have to add current availability and popularity from its catalog and interaction data. Together they express conditions that text similarity cannot recover: Interstellar should not rank first for a viewer who cannot stream it, and a request for a space film under 90 minutes should favor a shorter plausible match. Deciding how much each feature should matter, and how its value changes in combination with other evidence, eventually requires more than a fixed additive boost.
Before a model can learn those combinations, each categorical detail needs an explicit role. A hard requirement such as language, regional availability, or subscription entitlement should filter films before ranking. A preference such as genre, director, or runtime may justify a request-conditioned boost when mismatches remain eligible, or enter a learned ranker when its effect depends on other evidence. Recording the category’s source, allowed values, default for missing data, and intended query classes makes those choices testable on the slices where the feature should help, rather than letting a broad average hide damage to a minority category.
Correct bias in behavioral labels
The behavioral feature now has a place in the ranking decision, but its values come from results the current ranker chose to expose. Click logs record the query, the displayed films, their positions, and the viewer’s subsequent actions, so they appear to provide relevance grades at scale. A click occurs only after the system has decided which films to show and where to place them, however. Treating raw click-through rate, clicks divided by displayed results, as relevance therefore carries the previous ranker’s decisions into the next model.
The first distortion is position bias, the tendency for a result’s position to affect the chance that a user notices and clicks it. Results near the top receive more attention, so they collect more clicks even when lower results would be equally useful. Dividing clicks by all impressions, meaning every rendered result, does not correct that imbalance because rendering does not prove examination. A Simplified Dynamic Bayesian Network (SDBN) is a click model that is a probabilistic account of how examination, attraction, clicks, and satisfaction produce the observed event sequence. Its simplified examination rule uses a narrower denominator: within one search session, it treats results at or above the last click as examined and excludes results below that point.
Under this model, a click at rank 19 carries evidence because the user examined deeply enough to reach it. The grade is based on clicks among estimated examinations rather than among every rendered result. A search session here begins with one query and includes its result interactions until the query changes or the search ends. The assumption remains imperfect because a zero-click session reveals no last examined position. Scroll, hover, or dwell telemetry, interface events recorded for later analysis, can provide additional evidence, but each event still needs an explicit interpretation.
Estimating examination gives the rank-19 click a more defensible denominator, but it does not tell the team how much to trust the resulting rate. One click from one estimated examination and 100 clicks from 100 examinations both produce 1.0. The first may be an accident; the second describes a pattern. The behavioral grade therefore needs to carry its uncertainty into training rather than allowing these two histories to exert the same influence.
A Beta distribution makes that uncertainty explicit. Its parameters and form a prior, the system’s starting assumption about click probability before the current film has gathered enough observations. After c clicks across e estimated examinations, Bayesian inference combines that prior with the observed history. The resulting posterior mean, the updated estimate of click probability, is
For a film with little history, the prior keeps one click from producing an extreme grade. As examinations accumulate, the observations exert more influence on the posterior mean, while the posterior variance falls to show that less uncertainty remains. A learning-to-rank model can turn that confidence into a sample weight, a multiplier controlling how strongly the query-film example contributes to the learning objective. A large, consistent history can then influence training more than the isolated rank-19 click. Because the prior controls how quickly sparse evidence moves away from its starting grade, its parameters must be fitted to the traffic and outcome rates of the film-search service.
The team now has a grade and a measure of confidence, but neither establishes that the interpretation of the clicks is useful. The judgment list remains a model of viewer behavior rather than a neutral record. Before training a ranker, the team can apply the derived grades directly as a temporary boost and evaluate them in an A/B test, a controlled experiment that compares the changed ranking with the existing one on separate traffic groups. If the boost makes the ordering worse, a learned model will usually hide the faulty interpretation rather than correct it.
Even a successful test leaves one boundary intact: examination correction can interpret only films the current ranker chose to show. A promising candidate outside the displayed window receives no examinations, so it cannot accumulate the clicks needed to challenge the ordering. This presentation bias can make a model trained on behavior reproduce the same ranking that generated its labels.
Controlled exploration creates some of the missing evidence by exposing a small, guarded share of traffic to promising but uncertain films. Those observations extend the training data into parts of the feature space, combinations of feature values, that the current ranking rarely exposes. More formal approaches use propensity estimates or counterfactual objectives to account for the chance that a result was observed; the primary papers on jointly learning a ranker and propensity model and a general counterfactual learning-to-rank framework develop those methods and their assumptions. None makes clicks self-explanatory, but each gives the next training round evidence from beyond the old ranker’s preferred results.
Learn ranking weights from judgments
Learning to rank turns these judgments into a supervised learning problem. Its training data groups films by query and assigns each one a relevance grade. For every query-film pair, the system records features such as field-level BM25 scores, embedding similarity, popularity, release year, availability, runtime, or language match. A semantic score from a later model can join them when the latency budget permits it. The learned ranker estimates how these signals should interact instead of relying on the equal hand-set weights introduced in Chapter 1.
The direct test in the preceding section determines whether the behavioral grades carry useful evidence before a model obscures their effect. If they improve the ordering, a learned ranker can combine that evidence with other features and generalize beyond the query-film pairs that accumulated enough interactions. The next decision is what kind of error the training objective should reward the model for correcting.
The learning objective determines what the model treats as an error. A pointwise objective predicts each film’s grade independently, even though the search system ultimately needs an ordering. A pairwise objective instead learns preferences between two films judged for the same query: film A should score above film B. At inference time, the model still assigns each film one score, but those scores were learned from within-query comparisons.
A pairwise objective knows that Interstellar should outrank a weaker film, but it does not by itself express where that correction would help the viewer most. Moving Interstellar from rank 20 to 19 leaves it buried, while moving it from rank 2 to 1 changes the first result. A listwise objective accounts for that difference by evaluating the order as a whole. LambdaMART, a mature approach built from gradient-boosted decision trees, gives more weight to pairwise corrections that would produce a larger improvement in a list metric such as nDCG. The training signal therefore concentrates on the positions that most change the ranking a viewer experiences.
Once the objective identifies which ordering mistakes matter, the trees can learn which evidence should correct them. Runtime can carry decisive weight for “space movie under 90 minutes” and almost none for the
That narrower set is trustworthy only if the evaluation measures decisions the model did not see during training. Splitting train and test data by query keeps every example derived from the
Apply cross-encoder reranking
The intermediate ranker leaves a smaller set whose films have survived retrieval scores, behavioral evidence, and domain features. Those signals can place an available film with the right runtime above an ineligible or inconvenient alternative, but they may still miss a relationship expressed across the query and summary. For Interstellar, the unresolved comparison is whether one film joins the request’s clues about space, a father, and slow aging more completely than another. Answering it requires the pipeline to spend more attention on the text of each surviving pair.
A hybrid retriever cannot perform that comparison directly because it encodes the query and each document separately, then reduces the pair to one similarity score. A cross-encoder instead processes the query and document together, allowing attention to connect words and phrases across both texts before producing a relevance score. The joint input can preserve relationships that separate representations lose, but it also prevents the document-side computation from being reused across queries. Every surviving candidate requires another model inference, which places the cross-encoder near the end of the pipeline.
The instrument below makes that cost boundary part of the test. It takes the fused RRF ranking from Chapter 4, selects only the top 5, 10, or 20 candidates, and rescores those query-document pairs. Each row traces one film from its RRF position to its cross-encoder position, while a star marks a labeled relevant result. Changing the window determines which candidates receive the joint comparison; the model scores every film admitted to that window in the same way.
Begin with “M3GAN” or “shark attacks a beach town”. In both cases, the reranker preserves the labeled film near the top. Then select “space movie where the dad ages slowly”. With a candidate window of 10, the model moves Boyhood above Interstellar, favoring the aging theme while losing the request’s space context. For “robot falls in love”, it promotes Vertigo and demotes Her. The model used here was trained for general web passage ranking, so these movie-query failures are evidence of training mismatch rather than evidence that joint scoring is inherently better or worse.
Choose a query and candidate window. The cross-encoder scores those query-film pairs, reorders them, and each row traces a film from its fused RRF rank to its new rank. Select a film to inspect its movement; a star marks a labeled relevant result.
The model’s extra attention does not guarantee a better judgment. Its score cannot reveal whether the training data taught the relationships this film corpus requires, so the team must measure the reranked output on representative queries just as it measured candidate coverage at the window boundary. A domain-appropriate reranker or fine-tuning on labeled film pairs may repair the mismatch. If joint scoring continues to degrade the baseline, the evidence supports leaving the cross-encoder out rather than paying more latency for a worse order.
Even a well-matched cross-encoder judges only the text it receives. Availability, release year, popularity, runtime, and language remain invisible unless they are encoded into that input, and forcing structured features into prose makes their influence harder to control. Its semantic score therefore reaches the next boundary as one part of the decision. The pipeline still needs a final order that preserves both the cross-encoder’s textual judgment and the structured evidence attached to each survivor.
Compose the ranking pipeline
Producing that final order requires a second kind of fusion. Chapter 4 combined lexical and dense rankings so a film could survive when either representation found useful evidence. Ranking now combines the evidence attached to those survivors: retrieval scores, behavioral history, domain constraints, learned feature interactions, and the cross-encoder’s joint query-document judgment. Retrieval fusion decides what remains available to rank; ranking fusion decides what deserves attention at the top of the list.
The
As these values converge on one order, each signal needs one owner. A click-derived value can act as a direct boost for frequent queries or enter the learned ranker as a feature, but applying the same aggregation in both places counts the behavior twice. The duplication makes its effective weight difficult to interpret and strengthens the feedback loop that produced the signal. Retrieval and model scores require the same discipline: record where each value enters, how it is normalized, and which phase is allowed to change its influence.
When the final composition places Interstellar ahead of weaker alternatives without exceeding the latency budget, ranking has completed the decision that retrieval left open. Its output is a short, ordered set of documents, together with the recorded scores and feature values needed to inspect that order. A person may read the list directly, or a language model may receive it as evidence. In either case, ranking has decided which sources receive the limited attention available next. Chapter 6 examines how generation should preserve their claims and provenance, cite them, and decline to answer when the ranked candidates do not support a response.