Stage 3: Retrieval

A diagnostic reference. Read while optimizing an existing retrieval-augmented system. Each section serves either a human diagnosing the system or an agent interrogating it.

Concept

The retrieval stage produces a ranked candidate set of documents (or chunks, or passages) for a given query. It is the first online stage that reads the corpus, and its output is the substrate every downstream stage operates on: ranking and fusion rescore the candidates, generation grounds an answer in them, evaluation measures their quality.

Retrieval is operationally a choice between three families of techniques, often combined:

• Lexical retrieval: term-matching over an inverted index, scored by BM25 or a variant. Excels at exact-token needs (names, IDs, technical terms). The historical foundation of search; every modern technique either supplements or rests on top of it.
• Dense vector retrieval: query and documents embedded into a shared vector space; nearest neighbors returned by an approximate nearest neighbor (ANN) index. Excels at semantic equivalence (synonyms, paraphrases, multimodal). Systematically weak on exact-token needs.
• Hybrid retrieval: both, fused. The options form a spectrum rather than a binary choice: pure lexical → learned-sparse (SPLADE, semantic knowledge graphs) → sparse retrieval with dense reranking → fused candidate sets (RRF or weighted additive) → late interaction (ColBERT, multi-vector) → pure dense.

Two cross-cutting decisions shape every retrieval implementation: which ANN index family (HNSW or IVF, mainly) and what compression (scalar, binary, product, or matryoshka quantization, in various combinations). Both have their own operational profiles (recall, latency, memory, build cost, filter behavior), and the right combination depends on the corpus, the query mix, and the latency budget more than on any leaderboard ranking.

A unifying pattern threads through the entire stage: over-request and rerank. Every cost-cut (quantization, ANN approximation, learned-sparse) trades some recall for cheaper candidate generation, and every such cut is paired with over-fetching and rescoring the top candidates with a higher-fidelity scorer. This pattern is universal enough to be its own diagnostic principle.

What good looks like at this stage: for the specific corpus and query distribution this system serves, the relevant items land in the top-K candidate set at small enough K that downstream stages can afford to rerank. Leaderboard recall numbers are a poor proxy; what matters is recall on the system’s own queries rather than on TREC or MS Marco. Calibrate diagnostic effort against the system’s own labeled queries rather than published benchmarks.

Symptom catalog

recall_metric_ambiguity: Confusion between two distinct recall measurements

• Observable signal: The team quotes a recall figure (“our quantization achieves 98% recall,” “binary gives 60%”) without naming the reference ranking. User-facing relevance is poor despite high reported recall numbers.
• Frequency: Very common; the source course treats this disambiguation as the most important clarification in lesson 8.
• Cost of leaving unaddressed: Misdirected optimization effort. The team tunes compression parameters when the actual bottleneck is the embedding model, query understanding, or ranking.

The two measurements:

Recall measurement	What it answers	When it’s useful
Recall vs. full-precision ranking	”How well does the compressed/approximate search approximate the uncompressed/exact search?”	Tuning quantization, ANN parameters, ef_search
Recall vs. ground-truth relevance	”How well does the search find relevant documents for the user?”	Choosing embedding models, evaluating system quality

A retrieval pipeline can have 99% of the first and 30% of the second simultaneously. Quantization benchmarks measure the first. User-facing quality requires the second.

vector_dominance: Dense retrieval crowds out exact-term needs

• Observable signal: Queries with rare or exact tokens (product SKUs, person names, technical IDs, specific model numbers) systematically miss obvious matches. The false-positive rate is elevated: results look plausible and are wrong.
• Frequency: Common in any system that defaults to dense-only or heavily-dense-weighted retrieval, particularly in e-commerce, legal, and medical domains.
• Cost of leaving unaddressed: Direct conversion impact in e-commerce; legal/medical correctness risk; user trust erosion.

The structural cause: dense embedding models summarize semantics into a few hundred to a few thousand dimensions. The summary necessarily lossy-compresses exact-token information. A query containing a rare token cannot be reliably distinguished from a query containing a related-but-different token at the embedding level. The fix is hybrid retrieval with a meaningful lexical component; better embeddings cannot supply the missing exact-token evidence.

score_distribution_mismatch: Naive fusion silently picks a winner

• Observable signal: Hybrid retrieval results are erratic. Some queries look great; others look like pure lexical (or pure dense). Score ranges across the two retrievers differ by orders of magnitude.
• Frequency: Common in first-pass hybrid implementations that compute final_score = bm25_score + cosine_similarity without normalization or rank-based fusion.
• Cost of leaving unaddressed: Unpredictable behavior; the team believes it is running hybrid search while effectively running a single retriever per query.

BM25 scores are unbounded (can range from 0 to 30+ depending on corpus and query); cosine similarities are typically [-1, 1] (or [0, 1] if non-negative embeddings). A naive sum makes BM25 dominate. The fix lives at Stage 4 (Ranking & Fusion): use rank-based fusion (RRF) or normalize before additive scoring.

underfetch_quantized: Compression recall cap, missing over-fetch

• Observable signal: A team adopts binary or aggressive scalar quantization, observes user-facing recall drop, and blames the embedding model or the corpus.
• Frequency: Common in any first implementation of quantization; the over-request-and-rerank pattern is easy to miss.
• Cost of leaving unaddressed: Recall permanently capped at the quantization’s intrinsic recall (~60% for binary; ~93% for int8 scalar). Compression looks bad; people back out without realizing it would have worked with the pattern in place.

The full pattern: quantize for the candidate phase, over-fetch (typically 2x to 10x the desired top-K), then rerank the over-fetched candidates with full-precision vectors stored on disk and read only at this rerank step. On a small lab dataset, binary + over-fetch recovers to ~100% recall. At very large scale, recovery is partial and the over-fetch ratio must grow.

hnsw_segment_merge_lag: Lucene HNSW operational tax

• Observable signal: Indexing throughput drops sharply during segment merges. Query latency is fine. Most visible on Elasticsearch, OpenSearch, and Solr deployments using HNSW vector fields.
• Frequency: Common on high-write workloads with HNSW vector indexes in Lucene-based engines.
• Cost of leaving unaddressed: Indexing backlog; stale index visible to users; in extreme cases, the indexer falls behind the input stream.

The structural cause: Lucene’s HNSW implementation requires graph rebuilds on segment merges. The cost is proportional to the merged segment’s graph density (M, ef_construction). Workarounds: tune the merge policy to reduce merge frequency, accept higher disk usage; or move to a non-Lucene engine for the vector path (Qdrant, Pinecone) and keep the lexical path on Elasticsearch/OpenSearch.

ivf_drift: Cluster centroids age out

• Observable signal: IVF-indexed retrieval recall degrades gradually over weeks or months on a corpus with meaningful new-document inflow. No sudden change; just a slow decline.
• Frequency: Common in IVF deployments on high-churn corpora (news, social, frequently-updated catalogs).
• Cost of leaving unaddressed: Silent recall decay. Not visible in standard latency/error monitoring; visible only on retrieval-quality regression tests if those exist.

The structural cause: IVF clusters are learned from the initial corpus. As new documents arrive, they are assigned to the existing nearest centroid, but the optimal centroid placement has shifted. Mitigation: periodic re-clustering. Frequency depends on churn; quarterly is a reasonable starting cadence for moderate-churn corpora.

hnsw_filter_dominance: Selective filters destroy HNSW recall

• Observable signal: A query with a strict filter (category=X, user_id=Y, date>Z) returns far fewer or worse results than the same query without the filter, on an HNSW-indexed corpus.
• Frequency: Common in multi-tenant, e-commerce, and time-sensitive systems where filters are routine.
• Cost of leaving unaddressed: Filtered queries silently underperform. Often missed because the team evaluates unfiltered retrieval quality.

The structural cause: HNSW filters post-traversal. The graph walk produces candidates, then the filter eliminates non-matches, possibly all of them. With selective filters, the post-filter candidate pool can be empty. IVF filters pre-traversal (filter on cluster membership before per-vector scoring), which sidesteps this. The choice between HNSW and IVF is dominated by filter selectivity in any system that filters heavily.

crossencoder_training_mismatch: Reranker domain doesn’t match corpus

• Observable signal: Cross-encoder reranking helps in a lab notebook (MS Marco-style data) but underperforms on the target corpus. Production relevance evaluations show the cross-encoder reordering candidates worse than the BM25 baseline.
• Frequency: Common in any system that adopts an off-the-shelf cross-encoder (typically cross-encoder/ms-marco-MiniLM-*) without domain evaluation.
• Cost of leaving unaddressed: A negative-value-add reranker; the team often does not notice the reranker is hurting because it is measured against the cross-encoder’s own training distribution.

The mitigation menu, in increasing investment:

• Substitute with a domain-fitting off-the-shelf model (e.g., a cross-encoder trained on biomedical, legal, e-commerce data).
• Finetune on a few thousand labeled query-document pairs from the target domain.
• Replace cross-encoder with a learning-to-rank (LTR) model that incorporates domain-specific features beyond semantic relevance.

crossencoder_recency_blind: Semantic-only ranking ignores time

• Observable signal: Newer content that is semantically equivalent (or slightly less semantically similar) to older content ranks below the older content. Users report results feeling stale.
• Frequency: Visible whenever recency is a genuine relevance signal: news, social media, technical documentation, software changelogs, e-commerce with seasonality.
• Cost of leaving unaddressed: Persistent staleness complaint; the team may iterate on the embedding model when the fix is structural.

Cross-encoders score query-document semantic relevance only. They have no notion of time. Recency is a non-semantic feature. The fix lives at Stage 4 (Ranking & Fusion): a learning-to-rank model that combines the cross-encoder’s semantic score with recency, popularity, and other non-semantic features; or a simple time-decay reweighting applied after cross-encoder scoring.

chunk_boundary_loss: Answers split across chunks

• Observable signal: Test queries with known-correct passages fail to retrieve those passages. Surrounding-but-incomplete chunks are returned instead. The corpus contains the answer; the chunking destroyed it.
• Frequency: Common in any system with arbitrary fixed-length chunking and no boundary evaluation.
• Cost of leaving unaddressed: Hard ceiling on retrieval quality; downstream stages cannot recover the lost information. Generation will hallucinate to fill the gap.

Two complementary mitigations:

• Extractive-QA-based chunk evaluation: define a small test set of (question, expected-answer-span) pairs; verify chunks contain complete spans. From the AI-Powered Search book’s Chapter 14 labs.
• Late chunking: embed the whole document at the token level first, then chunk, mean-pooling the token embeddings within each chunk. The chunk’s embedding carries cross-chunk context. Essentially free at storage and search time once supported by the embedding model.

unnormalized_relatedness: Unbounded scores break term expansion

• Observable signal: A team uses Elasticsearch’s significant_text aggregation (or OpenSearch’s equivalent) to find related terms for query expansion. The relatedness scores are unbounded; setting a threshold that works across queries is impossible.
• Frequency: Specific to teams pursuing query expansion via stock aggregations in Lucene-family engines.
• Cost of leaving unaddressed: Either the team hand-tunes thresholds per query (unscalable) or abandons query expansion (loses signal).

The structural cause: significant-terms-style aggregations score by a raw frequency-ratio statistic with no normalization. The Semantic Knowledge Graph (SKG) relatedness function is bounded to roughly [-1, 1] by design, with uninformative terms (stop words) auto-centering near zero. SKG’s relatedness is the structural fix. Apache Solr has first-class SKG with multi-hop traversal; OpenSearch and Elasticsearch have community single-hop ports via painless scripts.

How to inspect

By code pattern

Greppable signals that reveal retrieval-stage configuration choices.

Pattern / location	What you’re looking for	What suggests a problem
top_k, topk, k=, limit= in retrieval calls	The candidate-set size	Hard-coded small K (<20) without downstream over-fetch; or large K (>1000) with expensive downstream rerank
embedding_model, embed_model, or model-name strings (bge-, text-embedding-, ms-marco)	The chosen embedding model	Defaulted to a multilingual or general model in a domain-specific corpus; never re-evaluated against a labeled set
cross-encoder/ms-marco-	Off-the-shelf cross-encoder	Suggests crossencoder_training_mismatch if domain isn’t general web QA
index_type=HNSW, M=, ef_construction=, ef_search=	HNSW parameters	All defaults; no per-corpus tuning; or values copied from a tutorial
nlist=, nprobe=, IVF	IVF parameters	Never re-clustered after initial index build; nprobe=1 for production queries
int8, binary, quantize, pq in index config	Quantization setting	Quantization on but no over-fetch in the search path (look for the same top_k value used end-to-end)
chunk_size, chunk_overlap, RecursiveCharacterTextSplitter	Chunking config	Fixed-length splitting; zero overlap; no extractive-QA-based evaluation against a test set
bm25_score + cosine_similarity or similar additive fusion	Naive hybrid fusion	Score scales aren’t normalized; suggests score_distribution_mismatch
rrf, reciprocal_rank_fusion, 1.0 / (k + rank)	RRF	Default k=60 never tuned; or k value copied from a paper without held-out validation
if bm25_results: return; else: vector_results	Conditional fallback	Not hybrid; one retriever runs, the other only on empty results, defeating fusion benefits
Filter clauses (filter=, where=) on HNSW-indexed search	Filtered HNSW queries	If filters are selective, suggests hnsw_filter_dominance

By measurement

Metrics, logs, and traces to enable for retrieval diagnosis.

Measurement	How to compute	Healthy range / red-flag threshold
Recall@K vs. labeled relevance	Manual or LLM-as-judge labeled set of (query, relevant_doc) pairs; compute the fraction where relevant doc appears in top-K	Domain-dependent; target ≥80% recall@10 for most retrieval systems; <50% almost always indicates a structural problem
MRR (mean reciprocal rank)	For queries with one correct answer, average 1 / rank_of_correct	Lower is worse; benchmark against a stable test set as ground truth
nDCG@K	When relevance is graded (0-3); penalizes putting marginal items above highly-relevant ones	Used when relevance has degrees
Recall@K vs. full-precision baseline (quantization recall)	Run the same top-K query against full-precision and against quantized; measure overlap	Quantization-tuning metric; distinct from the labeled-relevance rows above
p50 / p95 retrieval latency	Per-component (lexical, vector, fusion, rerank) latency in traces	Establish baseline; alert on >2x regression
Top-K result distribution	Per-query: how many results from lexical, dense, hybrid?	All-from-one-retriever in a supposed-hybrid system suggests score_distribution_mismatch
Filter post-traversal candidate count (HNSW)	After ANN walk, count results remaining after filter	Frequent 0s on selective filters suggest hnsw_filter_dominance
Indexing throughput during merge windows	Documents/second during segment merges	Sharp drops indicate hnsw_segment_merge_lag
IVF cluster size distribution drift	Bucket count per centroid over time	Increasing variance over weeks suggests ivf_drift
Recall on filtered vs. unfiltered queries	Same query set, measure with and without filters	Significant gap suggests hnsw_filter_dominance

Diagnostic questions

Triage flow for an agent or engineer assessing this stage. Earlier questions are coarse; branches go deeper when answers are unclear or red-flag.

1. [interview] What does “good retrieval” mean for this system, in terms a user would recognize? Until this is named, no later metric is interpretable. Force a concrete answer: “users find what they searched for in the top 10 results 90% of the time,” “the system surfaces the correct legal citation in the top 5 for known case-law queries,” etc.

2. [code] What retrieval techniques are running, and how are their outputs combined? Branches:

   • Pure lexical: ask about query mix; suspect missed semantic equivalents.
   • Pure dense: ask about exact-token queries; suspect vector_dominance.
   • Hybrid: continue to fusion questions below.

3. [code, if hybrid] How are lexical and dense results fused? Branches:

   • Additive scoring without normalization: suspect score_distribution_mismatch; check score-scale assumptions.
   • RRF with default k: ask if k was ever validated against held-out queries.
   • Filter-then-rerank: ask about the filter; suspect hnsw_filter_dominance if HNSW.

4. [code] What is the top-K, and does the retrieval pipeline over-fetch before any rerank step? Branches:

   • Top-K returned directly without rerank: ask about candidate quality at that K; check whether the embedding/quantization recall ceiling caps results.
   • Over-fetch present: ask the ratio; <2x is suspicious for binary quantization.

5. [code] Which ANN index family, HNSW or IVF? Branches:

   • HNSW + selective filters: suspect hnsw_filter_dominance; consider IVF or hybrid pre-filter.
   • HNSW + Lucene engine + high-write workload: ask about segment-merge cost.
   • IVF + high-churn corpus: ask about re-clustering cadence; suspect ivf_drift.

6. [code, if quantization is on] What recall metric is being reported for the quantized index? Branches:

   • “Quantization recall” without naming the reference: distinguish recall_metric_ambiguity before any further tuning; nothing else can be interpreted until it is.
   • Recall vs. full-precision ranking + over-fetch present: quantization is correctly understood; quality questions live upstream (embedding model, query understanding).

7. [interview] When was the embedding model last evaluated against the system’s own labeled queries rather than benchmarks? Branches:

   • Never / on adoption only: suspect domain drift; run an evaluation now.
   • “It wins on MTEB”: ask whether MTEB queries resemble the system’s queries (they rarely do).

8. [code, if cross-encoder rerank present] What is the cross-encoder’s training data domain, and how does it compare to the corpus? Branches:

   • MS Marco trained, e-commerce corpus: suspect crossencoder_training_mismatch.
   • Recency matters in the domain: suspect crossencoder_recency_blind.

9. [interview] What is the chunking strategy, and has it been evaluated against known answer spans? Branches:

   • Fixed-length splitting, no eval: suspect chunk_boundary_loss; run an extractive-QA-based evaluation.
   • Late chunking: ask if the embedding model supports it correctly.

10. [code, if using term expansion] What primitive is producing expanded terms, and are its scores bounded? Branches:

    • significant_text aggregation, unbounded scores: suspect unnormalized_relatedness; consider SKG.

Interventions by likely cause

Ordered by frequency of cause. Cross-references to symptom IDs are in stage#symptom_id format.

Hypothesis	Supporting evidence	Intervention	Expected impact	Cost / risk
Quantized retrieval has no over-fetch step (retrieval#underfetch_quantized)	Same top_k used end-to-end; quantization on	Add over-fetch (start 2x for scalar int8, 5x to 10x for binary) and full-precision rerank	Recovers most of the recall lost to compression	Low: peak memory increases proportionally; otherwise minimal
Reported “recall” is quantization fidelity rather than relevance (retrieval#recall_metric_ambiguity)	Recall number cited without naming the reference	Re-measure recall against a labeled relevance set; separately measure quantization fidelity	Aligns optimization effort with the real bottleneck (often upstream: embedding model or query understanding)	Low: measurement work
Hybrid fusion uses naive additive scoring (retrieval#score_distribution_mismatch)	Code shows bm25 + cosine or similar; per-query winner is consistent (always lexical or always dense)	Switch to RRF, or normalize scores (z-score per query) before additive fusion	Stabilizes hybrid behavior across query types	Low: fusion swap is local
Pure dense retrieval; queries need exact tokens (retrieval#vector_dominance)	Domain has rare tokens (SKUs, names, IDs); recall on those queries is low	Introduce a lexical component (BM25) and fuse via RRF or filter-then-rerank	Substantial improvement on exact-token queries; moderate broader gain	Medium: adds an index and a fusion step
Cross-encoder trained on a different domain (retrieval#crossencoder_training_mismatch)	Off-the-shelf cross-encoder; lab eval positive, production eval negative	Substitute with domain-matched model; or finetune on a few thousand labeled pairs	Cross-encoder begins helping rather than hurting	Medium: finetuning needs data and infra
HNSW with selective filters (retrieval#hnsw_filter_dominance)	Filter selectivity high; HNSW post-filter candidate counts near 0	Move that path to IVF; or use a hybrid filter strategy (BM25 with filter as pre-filter, vector for unfiltered)	Recall on filtered queries recovers	Medium: index family change is operationally significant
Chunk strategy splits answer spans (retrieval#chunk_boundary_loss)	Known-correct passages not retrieved; surrounding chunks returned	Move to late chunking; or evaluate chunks with extractive-QA before deployment; or use overlapping chunks (typically 100-200 token overlap)	Eliminates a class of avoidable retrieval failures	Low to medium: re-index required; storage cost negligible
Cross-encoder ignores recency (retrieval#crossencoder_recency_blind)	Newer content equivalent to older content ranks below	Add LTR layer with recency feature, or apply time-decay reweighting after cross-encoder	Resolves stale-results complaint	Medium: LTR is its own subsystem
IVF centroids stale on high-churn corpus (retrieval#ivf_drift)	Recall degrading over weeks; corpus is high-churn	Re-cluster centroids; establish a periodic re-clustering cadence (e.g., quarterly)	Recovers recall to baseline	Low: operational; minimal latency impact
HNSW segment merges saturate indexing (retrieval#hnsw_segment_merge_lag)	Lucene engine; indexing throughput drops on merges	Tune merge policy (larger segments, less frequent merges); split vector path to a non-Lucene engine	Smoother indexing throughput	High: operational architecture change
Term expansion uses unbounded relatedness scores (retrieval#unnormalized_relatedness)	Significant-text aggregation in use; thresholds don’t generalize across queries	Switch to SKG (Solr first-class; OpenSearch/ES via painless port); or normalize scores per query	Stable, tunable term expansion	Medium: primitive swap
Embedding model is general-purpose; corpus is domain-specific (cross-cutting: embedding_finetuning)	Pure-dense or hybrid retrieval; quality low on domain queries	Substitute with domain-matched embedding model; or finetune on domain labeled pairs	Often substantial; the highest-leverage retrieval improvement in domain-specific systems	High: finetuning infrastructure and ongoing maintenance