Stage 1: Corpus and Indexing

A diagnostic reference. Read while optimizing an existing retrieval-augmented system. Each section serves either a human diagnosing the system or an agent interrogating it.

Concept

The corpus and indexing stage is everything that happens to documents before any query arrives. It ingests raw content, runs text through an analysis chain (tokenization, lowercasing, stemming or lemmatization), splits documents into chunks, computes embeddings, and builds the structures retrieval will read: the inverted index with its posting lists and term statistics, dense ANN indexes (HNSW or IVF, possibly quantized), and multi-vector indexes for late-interaction models. All of it runs offline. Cost paid here is paid once per document; cost avoided here is paid on every query, forever.

Two properties make this stage diagnostically distinctive. First, its failures are silent. A misconfigured analyzer, a chunk boundary through an answer span, a discarded full-precision vector: none of these errors out. Retrieval simply cannot find what the index never faithfully stored, and the evidence surfaces downstream as “bad relevance” with no pointer back to the cause. Second, its decisions set hard ceilings. No amount of query-time tuning recovers a token the stemmer destroyed or semantic detail the quantizer threw away without a full-precision copy to rerank against. When diagnosing a retrieval system top to bottom, this stage is where ceilings are checked before knobs are turned.

The stage also produces one asset for free: a search engine’s inverted index (term to documents) and forward index (document to terms) together form a bipartite graph, the semantic knowledge graph, which exists the moment documents are indexed with no extra construction step. Its consumption belongs to query understanding and retrieval; its existence is a property of this stage.

One production rule ties the stage together: a transform is part of the artifact contract. Version the tokenizer, chunker, normalization rule, embedding model, and vector dimension with every index. The serving path must use the compatible query-side transforms. Reusing one implementation is safer than asking training, ingestion, and serving teams to reproduce the same rule three times.

What good looks like at this stage: the index preserves everything the system’s query mix will need. High-value exact tokens survive analysis, answer spans survive chunking, semantic detail survives compression (or full-precision copies survive for recovery), every stored vector matches the query-time embedding model, and the structures stay fresh as the corpus churns.

Symptom catalog

analyzer_mismatch: Index and query analysis chains disagree

• Observable signal: Queries containing terms that verifiably exist in documents return nothing, or matching works for some morphological variants and not others. Often appears after an engine upgrade, a field remapping, a new query path that bypasses the standard analyzer, or a migration between engines with different defaults.
• Frequency: Common, and disproportionately common after infrastructure changes.
• Cost of leaving unaddressed: A class of queries silently returns nothing or garbage. Because the failure is per-term rather than global, aggregate metrics dip rather than crash, and the problem survives casual inspection.

The structural cause: lexical matching happens in token space rather than text space. A term stemmed one way at index time and another way at query time lands in different posting-list keys and never intersects. The source course notes that query-time tokenization is almost always, but not necessarily, the same as index-time, which cuts both ways: asymmetry is occasionally intentional, so engines allow it, and that permission is what makes accidents possible.

lossy_analysis_chain: Aggressive analysis destroys distinctions

• Observable signal: Distinct terms conflate (a product name stemmed into a common word; two different technical terms folding to one stem), or rare-but-critical tokens disappear because a stopword list or aggressive filter consumed them. Exact-token queries misfire in ways that get misdiagnosed as relevance problems.
• Frequency: Common in systems running default analyzers built for general web text over domain corpora (catalogs, legal, medical, code).
• Cost of leaving unaddressed: Permanent precision loss on exact-term queries, the query class where lexical retrieval is supposed to be the reliable component. In e-commerce this converts directly to lost conversions.

Stemming is heuristic by construction, and the recall-precision trade it makes is corpus-dependent: heavy normalization helps morphological variants and hurts identifiers. The information destroyed at index time cannot be reconstructed at query time; the fix is a re-index with a corrected chain.

chunk_explosion: Chunking multiplies the index

• Observable signal: After adopting chunk-level indexing, document count multiplies by 100x to 10,000x. Vector memory, index size, and ANN build times balloon; infrastructure cost becomes the dominant conversation. The source course’s worked scale: 50M documents at 2K chunks each is 100B vectors.
• Frequency: Very common in any RAG system that chunks long documents naively, one indexed document per chunk with parent metadata copied onto each.
• Cost of leaving unaddressed: Memory and storage costs grow superlinearly with corpus growth; ANN build and merge times degrade; at the far end, the system hits scale walls only specialized engines address.

The mitigation menu from lesson 6, in increasing invasiveness:

• Trust postings compression for the metadata. Duplicated parent text across chunk documents is cheaper than feared; a posting list stores a count rather than a copy.
• Field collapse / faceting at query time over flat chunk documents, to de-duplicate at result rendering.
• Parent-child block joins (Solr): chunks as children, retrieval returns parents via aggregation.
• Multi-valued vectors on parent documents (now native in Solr/Lucene): one document, many vectors, no join.
• Compression for the vectors themselves: the quantization stack (see full_precision_discarded below for the constraint that comes with it).

The source course is candid that 100B-vector scale has no clean answer; that boundary belongs to specialized engines (TurboPuffer, Vespa are the named examples).

stale_embeddings_model_swap: Vectors from a different model than the queries

• Observable signal: Vector retrieval quality collapses after an embedding model change or finetune deployment, or degrades in patches where new documents behave differently from old ones. Similarity scores look plausible individually but rankings are incoherent.
• Frequency: Common wherever embedding models are updated, and nearly guaranteed during partial migrations or interrupted backfills.
• Cost of leaving unaddressed: The dense retrieval path returns noise for the affected slice of the corpus. Mixed-version indexes are the insidious case: quality varies by document age, which defeats spot checks.

The structural cause follows from the bi-encoder premise the source course teaches: document meaning is fixed at index time by whatever model embedded it. Query vectors from a new model and document vectors from an old one occupy different spaces; cosine similarity across spaces means nothing. Finetuning (which the source course recommends trying almost always; see the embedding_finetuning cross-cutting doc) changes the space just as surely as a model swap. The consequence: re-embed the corpus on any model change, and never serve mixed versions.

full_precision_discarded: Quantized index with no recovery path

• Observable signal: A quantized vector index performs at its raw compressed recall (roughly 60% for binary, against full-precision ranking) and no over-fetch configuration improves it. Attempts to add a rerank step discover there is nothing to rerank against.
• Frequency: Occasional, but decisive when present; typically a storage-cost decision made without awareness of the recovery pattern.
• Cost of leaving unaddressed: A permanent recall ceiling, and the only exit is re-embedding the corpus (or recovering originals from another system).

The universal pattern that makes aggressive quantization safe is over-fetch and rerank: quantized search produces candidates cheaply, then the top candidates are rescored against full-precision vectors stored on disk and touched only at that step. Discarding the full-precision copies at index time removes the pattern’s precondition. Lesson 9’s mixed-precision layouts make the intended shape concrete: bit vectors hot in memory for traversal, float vectors cold on disk for rerank. The related recall arithmetic and the metric confusion around it live at retrieval#underfetch_quantized and retrieval#recall_metric_ambiguity; the storage obligation lives here.

bit_quantization_small_model: One bit per dimension, too few dimensions

• Observable signal: Binary quantization is enabled and vector retrieval lands below the plain BM25 baseline on the same corpus.
• Frequency: Occasional; appears when compression advice is applied without its model-size condition.
• Cost of leaving unaddressed: The vector path costs money to run and subtracts value; a zero-tuning BM25 index would beat it.

Small models do not have enough dimensions to survive 1-bit compression. The source course’s benchmark (lesson 9): a ~384-dim model at bit precision fell below the BM25 baseline, while a 768-dim model at bit precision beat the small model at bfloat16 using roughly 8x less storage. The variable to optimize is the product of dimensions and precision rather than either alone; more dimensions at lower precision often wins. Related: bfloat16 halves storage at essentially zero recall loss regardless of model size, and is the safe default before any lossier step.

hnsw_merge_rebuild_cost: Segment merges rebuild the graph

• Observable signal: Indexing throughput drops sharply during segment merges on Elasticsearch, OpenSearch, or Solr deployments with HNSW vector fields. Query latency stays healthy. On high-write workloads the indexer falls behind its input stream and users see stale results.
• Frequency: Common on high-write Lucene-family deployments with vector fields; invisible on read-mostly corpora.
• Cost of leaving unaddressed: Growing indexing backlog and index staleness. Because query-side dashboards show nothing, the problem is routinely discovered late.

Lucene’s HNSW implementation rebuilds the graph when segments merge, at a cost proportional to graph density (M, ef_construction). Interventions: tune the merge policy toward larger, less frequent merges and accept the disk overhead; reduce graph density if the recall budget allows; or split the vector path to a non-Lucene engine and keep lexical on Lucene. The retrieval-facing view of the same phenomenon is cataloged at retrieval#hnsw_segment_merge_lag; the root cause and every intervention are indexing-side.

Created here, observed downstream

Two failure modes originate at this stage but are cataloged in the Retrieval chapter, where they are observed. Diagnose there, fix here:

• retrieval#chunk_boundary_loss: answer spans split across chunk boundaries; surrounding-but-incomplete chunks retrieved instead. The chunking strategy chosen here is the cause; the fixes (extractive-QA span evaluation, late chunking, overlap) are applied here. Late chunking deserves emphasis because of its cost profile: embed the whole document at token level, chunk afterward, mean-pool per chunk. Search cost, storage, and indexing cost stay roughly the same, so it is essentially a free upgrade wherever the embedding model exposes token-level outputs.
• retrieval#ivf_drift: IVF centroids learned at build time age out on a high-churn corpus. The periodic re-clustering that fixes it is an indexing-side batch job.

How to inspect

By code pattern

Configuration reads and greps that reveal indexing-stage decisions.

Pattern / location	What you’re looking for	What suggests a problem
Analyzer / mapping definitions (analyzer, tokenizer, filter, field mappings)	The index-time and query-time chains per field	Different chains on the two sides without a documented reason; defaults never reviewed for the domain
stopwords, stemmer, porter, snowball in analysis config	Aggressiveness of normalization	Heavy stemming plus stopword removal on a corpus full of IDs, names, or citations
chunk_size, chunk_overlap, RecursiveCharacterTextSplitter, custom splitters	The chunking strategy	Fixed-length splitting, zero overlap, no evaluation artifact anywhere in the repo
Embedding calls at ingest (embed, encode, model-name strings like bge-, text-embedding-)	Which model produced stored vectors	No model version recorded alongside vectors; ingest and query paths referencing different model names or versions
Token-level embedding usage (token_outputs, mean-pooling over spans)	Late chunking	Its absence is not a bug, but chunks embedded independently plus known boundary complaints point at the standard-chunking limitation
Index config: int8, binary, bbq, quantize, pq, bfloat16	Vector precision decisions	Quantization enabled with no full-precision or bfloat16 copy retained; binary quantization with a small-dimension model
Vector field definitions (dims, dimension)	Model size vs precision	dims around 384 combined with 1-bit quantization
Merge policy settings (merge.policy, segment size settings) on Lucene engines	Merge behavior for HNSW fields	Defaults on a high-write workload with vector fields
nlist, kmeans, centroid training jobs	IVF build and refresh	No scheduled re-clustering on a corpus with meaningful churn
Normalization at ingest (np.linalg.norm, normalize_embeddings=True)	L2 normalization discipline	Normalized on one side only; or normalized twice; or dot-product metric over unnormalized vectors
Backfill / migration scripts for embeddings	Re-embed discipline	Partial backfills; no check that all vectors share a model version
Artifact manifest or release metadata	Compatible schema, analyzer, chunker, embedding model, vector dimension, and index version	Components promoted independently with no compatibility check or rollback target

By measurement

Measurement	How to compute	Healthy range / red-flag threshold
Token-stream diff, index vs query analyzer	Run a sample of real queries and matching document text through both chains; diff the outputs	Any divergence not documented as intentional is a red flag
Exact-token survival rate	Pass the corpus’s high-value tokens (SKUs, names, citations) through the index-time chain; check each survives as a distinct term	100% expected; any conflation or loss on this set is a finding
Answer-span survival rate	Extractive-QA test set of (question, expected-span) pairs; fraction of spans falling entirely inside a single chunk	No established threshold; treat sub-90% as material and a falling trend on re-chunk as a regression
Chunks per document / total vector count	Count from the index	Ratios in the hundreds-to-thousands per document signal chunk_explosion territory; project growth before it walls
Vector model-version coverage	If versions are recorded: distribution of model versions across stored vectors	Anything other than a single version serving queries is a red flag; if versions are not recorded, that absence is itself the finding
Artifact compatibility check	At deploy time, compare the serving query transform and index manifest	Any mismatch blocks promotion; a missing manifest is a release-process finding
Vector recall vs BM25 baseline	Same labeled query set against the vector path and an untuned BM25 index	Vector path below BM25 with binary quantization on suggests bit_quantization_small_model; below BM25 generally, suspect the model or stale_embeddings_model_swap
Quantized recall vs full-precision ranking	Overlap of top-K from quantized and full-precision search	Tuning metric for compression; if it cannot be computed because no full-precision vectors exist, that is full_precision_discarded
Indexing throughput across merge windows	Documents/second over time, annotated with segment-merge events	Sharp periodic drops coinciding with merges indicate hnsw_merge_rebuild_cost
Index freshness lag	Time from document write to searchability	Growing lag on high-write workloads corroborates the merge problem
Storage per document	Index size divided by source document count, tracked over time	Step changes after chunking or precision changes; useful for validating quantization actually landed

Diagnostic questions

Triage flow for an agent or engineer assessing this stage. Earlier questions are coarse; branches go deeper on red flags.

1. [interview] When retrieval misses, does anyone check whether the index could have found the answer in principle? This stage’s failures masquerade as relevance problems. Establish the habit of asking “is it findable?” before “is it ranked well?”

2. [code] Are the index-time and query-time analysis chains identical for every searched field? Branches:

   • Different chains, undocumented: suspect analyzer_mismatch; run the token-stream diff.
   • Identical but default: check the defaults against the domain; run the exact-token survival test for lossy_analysis_chain.
   • Recent engine upgrade or migration: re-verify both, defaults change between versions.

3. [interview] What is the chunking strategy, and what evidence supports it? Branches:

   • Fixed-length, no evaluation: run an extractive-QA span test; suspect retrieval#chunk_boundary_loss.
   • Chunks embedded independently and the model supports token-level output: late chunking is an essentially free upgrade.
   • Chunk counts per document in the hundreds or more: continue to the explosion questions.

4. [code, if chunking] How are chunks stored, and what is the total vector count trajectory? Branches:

   • One flat document per chunk with copied metadata: fine at small scale (postings compression absorbs the text); check vector count projections.
   • Vector count heading past what memory affords: work the chunk_explosion menu (joins, multi-valued vectors, collapse, quantization).

5. [code] Which embedding model and version produced the stored vectors, and does the query path use the same one? Branches:

   • Version not recorded anywhere: flag it; then audit by behavior (compare retrieval quality on old vs new documents).
   • Model changed or finetuned since last full re-embed: stale_embeddings_model_swap until proven otherwise.
   • Multilingual corpus on an English model, or documents exceeding the model’s context window: model-fit problem, upstream of everything else here.

6. [code, if quantization is on] What precision is stored, and does a full-precision copy exist? Branches:

   • No full-precision (or bfloat16) copy retained: full_precision_discarded; the over-fetch pattern at Stage 3 cannot function.
   • Binary quantization on a model around 384 dims: bit_quantization_small_model; compare against BM25 baseline.
   • Float32 everywhere and storage is a pain point: bfloat16 is the free first move.
   • Codebook methods (PQ) on a real-time-update engine: check the engine actually supports training-pass compression; real-time-by-design engines cannot.

7. [code] Lucene-family engine with HNSW fields: how does indexing throughput behave during merges? Branches:

   • Sharp drops on a high-write workload: hnsw_merge_rebuild_cost; look at merge policy, graph density, or splitting the vector path.
   • Read-mostly corpus: deprioritize.

8. [code, if IVF] When were centroids last recomputed, relative to corpus churn? Branches:

   • Never, on a churning corpus: the fix for retrieval#ivf_drift is a re-clustering job owned here; establish a cadence.

9. [interview] When the corpus or models change, what is the recompute story? The general form of questions 5 and 8: every derived artifact at this stage (vectors, centroids, graphs, cluster assignments) is a snapshot of some input. List each artifact, name its inputs, and check something recomputes it when inputs change. Anything with no recompute path is a future incident.

Interventions by likely cause

Ordered roughly by frequency of cause. Interventions marked [re-index] require reprocessing the corpus; that cost dominates the decision between them.

Hypothesis	Supporting evidence	Intervention	Expected impact	Cost / risk
Index and query analyzers diverged (corpus_and_indexing#analyzer_mismatch)	Token-stream diff shows divergence; failures date to an upgrade or migration	Align the chains; add the token-stream diff to CI for analyzer changes	Restores matching for the affected term classes	Low if aligning query side; [re-index] if the index side must change
Analysis chain too aggressive for the domain (corpus_and_indexing#lossy_analysis_chain)	Exact-token survival test shows conflation or loss on high-value tokens	Lighter stemmer or per-field chains (light analysis on ID/name fields, fuller on prose); prune stopword list	Recovers exact-token precision where lexical search should be reliable	Medium; [re-index]
Chunking splits answer spans (retrieval#chunk_boundary_loss, root cause here)	Extractive-QA span-survival rate low; known-answer queries return neighboring chunks	Adopt late chunking if the model supports token-level output; otherwise re-chunk with boundary-aware splitting or overlap, validated by the span test	Removes a hard ceiling on retrieval quality	Low to medium; [re-index] of chunks and vectors
Embedding model changed without full re-embed (corpus_and_indexing#stale_embeddings_model_swap)	Mixed model versions in the index, or quality split by document age after a model/finetune deployment	Full re-embed with the current model; record model version per vector; serve one version only, cut over atomically	Restores the dense path for the affected slice	Medium to high; [re-index] of all vectors; batch embedding cost scales with corpus
Quantized index retains no full-precision vectors (corpus_and_indexing#full_precision_discarded)	Recall capped at compressed ceiling; nothing available to rerank against	Re-embed and store full-precision (or bfloat16) copies on disk alongside quantized vectors; then enable over-fetch + rerank at Stage 3	Unlocks the recovery pattern; recall recoverable toward full-precision levels	High; [re-index] of vectors plus storage increase
Binary quantization on a too-small model (corpus_and_indexing#bit_quantization_small_model)	Vector path below BM25 baseline; model around 384 dims at 1-bit	Move to a larger model before bit-quantizing, or back off to bfloat16/int8 for the small model	Vector path returns to adding value over lexical	Medium; larger model raises embed cost; [re-index]
Storing float32 with no compression at all	Vector memory/storage a cost pain point; no precision tuning ever attempted	bfloat16 as the first move (half the memory, essentially zero recall loss); consider int8/binary later with the retention rule above	Immediate storage relief at negligible quality risk	Low; [re-index] of vector storage, mechanical
HNSW graph rebuilds throttle indexing (corpus_and_indexing#hnsw_merge_rebuild_cost)	Throughput drops aligned with merge events on a Lucene engine	Tune merge policy (larger, less frequent merges); reduce graph density if recall budget allows; split vector path to a non-Lucene engine	Smoother indexing throughput, fresher index	Low for policy tuning; high for the engine split
Chunk-document count outgrowing infrastructure (corpus_and_indexing#chunk_explosion)	Vector count and index size ballooning; cost or build-time walls approaching	In order: verify metadata duplication is actually a problem (postings compression usually absorbs it); field collapse at query time; parent-child joins or multi-valued parent vectors; quantization stack for the vectors	Contains index growth; defers or removes the scale wall	Medium; storage-model changes are [re-index] and touch query code
IVF centroids stale on a churning corpus (retrieval#ivf_drift, fix owned here)	Recall declining over weeks; no re-cluster since build	Recompute centroids; schedule re-clustering on a cadence matched to churn	Recall returns to baseline	Low; periodic batch job
Embedding model poorly fit to corpus (language, length, domain) (cross-cutting: embedding_finetuning)	English model on multilingual corpus; documents exceeding context window; weak domain recall with correct versioning	Reselect via MTEB-style shortlist filtered by hard constraints, then evaluate on own labeled queries; consider finetuning (see cross-cutting doc)	Often the largest single quality lever at this stage	High; model change implies full re-embed [re-index]