Every instrument in this course searches the same corpus: 250 films, each represented by a title, release year, directors, origin countries, runtime, original language, genre labels, and a plot summary. Before a search system can judge which film is relevant, it must transform those records into evidence that retrieval can use. The system needs rules for what counts as a searchable term, how much text belongs in one retrievable unit, how meaning is represented, and how much detail the stored representation preserves.
Those decisions happen offline, before a user submits a query. Doing the work once per document keeps the online request path fast, but it also fixes what later stages will be able to see. If analysis removes an important distinction, chunking separates a question from its answer, or compression discards a useful signal, a ranking model or agent cannot recover the missing evidence at query time. This chapter examines how those indexing choices shape retrieval and how to test them before their failures appear elsewhere in the system.
Tokenization defines what can match
Before a lexical index can store a document, it runs the text through an analysis chain, an ordered sequence of transformations whose final output becomes the searchable representation of that document. The chain may split text on spaces or punctuation, normalize letter case, remove common words, and reduce related word forms to a shared base. The resulting searchable units are called tokens. A lexical search can match only the tokens produced by this process, so each transformation changes the evidence available at retrieval time.
The instrument below applies a small analysis chain to an editable document and the fixed query “x wing engine.” Both pass through the same steps. Toggle one step at a time and compare the resulting tokens, then watch which query tokens gain or lose an exact match.
At first, the query finds nothing in the sentence, even though the connection is obvious to a reader. Lowercasing removes the difference between X and x. Splitting on punctuation turns “X-Wing” into x and wing, allowing those two query terms to match. Stemming reduces engines to engine, which supplies the third match. Removing stopwords does not affect this particular query, but it changes which words the index stores.
That result depends on the user writing x wing as two words. If the query contained xwing, it would remain different from the indexed tokens x and wing. Supporting both forms requires another normalization rule or an explicit mapping supplied from domain knowledge. Analysis can encode known variations, but it cannot anticipate every way users may express the same concept.
The engine match also shows the tradeoff introduced by stemming. Removing endings by rule can improve recall, the system’s ability to retrieve relevant results, because related forms such as engine and engines produce the same token. A crude stemmer can also merge terms that should remain distinct: a suffix rule that reduces both universal and university to univers makes them indistinguishable in the index. Lemmatization uses a language’s vocabulary and morphology to map a word to its dictionary form. It makes a more informed choice than suffix stripping, but it requires language-specific resources. Either approach determines which variations match and which distinctions the index preserves.
Production systems often configure separate analysis chains for indexing and querying. The chains do not have to be identical, but they must produce compatible tokens. A query analyzer might expand a known synonym, for example, while the index stores only the canonical term. Problems arise when the difference is accidental. If the document chain stores x and wing while the query chain produces xwing, the lexical scorer receives no shared term to score. When an exact term or phrase cannot be retrieved, inspect the tokens produced on both sides before changing the ranking model.
▸Implement it: tokenize the course corpuscode/chapters/chapter_01.py
def tokenize(text: str) -> list[str]:
return TOKEN.findall(text.lower())What the index stores
Once analysis has produced tokens, a lexical search system records them in an inverted index. For each token, the index stores a list of the documents that contain it. This reverses the structure of the corpus: instead of scanning every document for a query term, the search engine can go directly from the term to a much smaller set of candidate documents.
The index also preserves evidence needed for ranking. It records how often a term appears in each document, the length of each document, and statistics across the corpus such as average document length. These values become inputs to BM25, a widely used keyword ranking function. Chapter 4 explains how BM25 combines those stored measurements for the current query. Much of what looks like query-time scoring therefore depends on evidence prepared offline.
Meaning-based retrieval stores a different representation. An embedding model converts each retrieval unit into a vector, a fixed-length list of numbers that represents features of its meaning. In a bi-encoder, the model encodes documents before search and encodes each query separately when it arrives. Retrieval then uses the distance between the query vector and stored document vectors as evidence of semantic similarity.
That comparison can use only the distinctions preserved by the embedding model. If the stored vector does not represent a detail that matters for a future request, query-time comparison cannot recover it from the original text. Model selection is therefore an indexing decision. Public benchmarks can narrow the options by language support, model size, license, and input length, but the final choice must be tested against the team’s own corpus and relevance judgments. Chapter 7 develops that evaluation process.
Stored vectors are also tied to the model version that produced them. A query encoded by a new or fine-tuned model cannot be compared reliably with documents encoded by the old model because the two sets of vectors may represent meaning differently. Changing the model requires re-encoding the corpus and rebuilding the vector index.
Choose the retrieval unit
A retrieval unit is the record that a search system scores and returns as a candidate. Each film summary in this course is about eighty words and already works as one unit. Real documents rarely arrive in such convenient pieces. A long manual may contain many unrelated procedures, and returning the entire manual gives the user too much material to inspect. Search systems therefore divide long documents into smaller units called chunks before indexing them.
Chunk size determines how much evidence travels together in one result. A small chunk can isolate the passage that closely matches a query, but it may omit definitions or qualifications from nearby text. A large chunk preserves more context, but unrelated material can weaken the retrieval signal and consume more of the system’s processing budget. The boundary between chunks introduces another risk: information needed for one answer may be divided across two candidates.
In the instrument below, each panel shows a retrievable chunk that contains at least part of the gold-underlined answer. At the default settings, the recommendation is divided between two panels. The overlap control repeats words from the end of one chunk at the beginning of the next, giving the complete passage another chance to travel together.
Chunks end after a fixed number of words, wherever that lands.
Maximum words stored in each chunk.
No words are repeated between adjacent chunks.
I bought the Meridian Alloy last spring for long winter rides on the coast road, and after four months of rough pavement it has quietly become my default training bike. For cold-weather rides I drop to 82 psi in the
rear and 78 up front, which keeps the ride compliant without risking pinch flats. At summer pressures the same wheels feel harsh once the temperature falls, so that adjustment matters more than any single component choice. The rest of the
Increasing the chunk size or adding overlap creates a chunk that holds the full answer. Before that change, retrieval could return either fragment, but neither contained the complete recommendation. Overlap duplicates some text and increases index size, but it reduces the chance that an arbitrary boundary separates information that belongs together.
The first two controls implement the two simplest strategies. Cutting the document every n words is fixed-size chunking; adding overlap turns it into a sliding window, which spends index size as insurance against a bad boundary. Both remain blind to the document itself, which is why the boundary landed inside the recommendation in the first place. Structure-aware strategies read the document before cutting it. A recursive splitter tries paragraph breaks first, then sentence breaks, then word boundaries, falling back a level only when a piece is still too large. A document-aware pipeline chooses those separators by format, such as Markdown headings, HTML sections, or the page and section structure of a manual. Switch the instrument’s split control to sentence boundaries and watch the result: the recommendation is a single sentence, so it stays intact at every size setting with no overlap at all.
Hierarchical chunking separates the unit that matches from the unit the reader receives. The system indexes small child chunks so retrieval can match precisely, then returns each child’s larger parent section so generation receives the surrounding definitions and qualifications. The price is a second layer of storage and a parent mapping that must survive index rebuilds.
The right strategy and settings depend on the documents and requests the system must handle. Build a small evaluation set with representative questions and known answer passages, then measure whether the chunker keeps those passages intact and whether retrieval returns them. Compare those results with the storage, latency, and context costs of each configuration. This turns chunk size and overlap into testable indexing choices rather than fixed defaults.
Late chunking offers another option for embedding-based retrieval. Instead of splitting the document first and embedding each chunk without its surroundings, the model processes the longer document at the token level before the token representations are grouped into chunk vectors. The resulting vectors can preserve context from outside each chunk, such as the subject of a pronoun or a definition from an earlier paragraph. The method requires an embedding model with suitable token-level output and enough input capacity for the source document, so it should be evaluated against conventional chunking on the same requests.
▸Implement it: split documents with overlapping windowscode/chunk.py
def chunk(words: list[str], size: int = 40, overlap: int = 10) -> list[list[str]]:
"""Fixed-size sliding window with overlap. Overlap is cheap insurance:
a boundary that lands inside an answer span destroys it for retrieval,
and overlapping windows give every span a second chance to sit whole
inside some chunk."""
step = max(size - overlap, 1)
return [words[i : i + size] for i in range(0, len(words), step)]Reduce vector storage without discarding evidence
An embedding stores one numerical value for every vector dimension. The 384-dimensional vectors for this course’s 250 films require less than half a megabyte when each value uses the 32-bit float32 format. At millions of chunks, however, those values consume substantial storage and memory bandwidth. The final indexing decision in this chapter is how much numerical precision retrieval needs and which higher-precision representation the system should retain.
Quantization reduces the number of bits used to represent each vector value. Converting float32 values to 16-bit bfloat16 halves their size. Scalar int8 quantization stores one 8-bit integer per dimension, reducing vector storage by 75 percent. Binary quantization stores one bit per dimension, reducing it by about 97 percent. Greater compression makes the index smaller and cheaper to search, but it removes more information from the vectors.
Measure that loss by comparing retrieval from the compressed index with retrieval from the full-precision vectors. For a set of representative queries, record how many full-precision nearest neighbors remain in the compressed candidate set at the same or a larger cutoff. This measures fidelity to the original vector search, which differs from recall against human relevance judgments. A compressed index can disagree with the full-precision ranking and still return useful results, while perfect agreement can preserve the mistakes of a poor embedding model. Evaluate both.
Binary-vector retrieval commonly compares vectors with Hamming distance, the number of bit positions at which they differ. This operation is compact and efficient, but its retrieval quality depends on the embedding model, the number of dimensions, and the data being searched. A configuration that works for one model or corpus may remove too much evidence from another, so compression level belongs in the same relevance evaluation as model and chunking choices.
A two-stage search can recover much of the lost ordering accuracy. First, retrieve more candidates than the user will see from the compressed index. Then rescore that larger candidate set with higher-precision vectors and keep the best results. The compressed representation supports broad, inexpensive retrieval, while the more precise representation makes the final distinctions. Chapter 5 applies this pattern to ranking and shows why the candidate window must be measured rather than assumed.
This recovery pattern requires the higher-precision vectors to remain available. They can live on disk rather than in the memory used by the first-stage index, but the rescoring step still needs to read them. If the indexing pipeline discards them, later stages cannot restore the missing precision without encoding the corpus again. Store the compressed vectors for broad retrieval and retain a compatible higher-precision copy for rescoring.
The index is a versioned artifact
An index is the output of a specific preparation pipeline. The same corpus will produce a different searchable artifact when the document schema, analysis chain, chunking rules, embedding model, or quantization settings change. A production index should therefore carry a version that identifies the corpus snapshot and every configuration or model needed to reproduce it.
That version also defines compatibility with the online path. The query analyzer must produce tokens compatible with the lexical index. Query embeddings must come from the same model version as the stored document embeddings. A rescoring stage must know which higher-precision vectors belong to the compressed index. Mixing artifacts from different versions can degrade retrieval without causing a request error.
Deploy index changes as coordinated releases. Build the new version alongside the one currently serving traffic, verify document counts and a representative set of queries, then switch the online path to the complete artifact. Keeping the previous version available makes rollback possible if production behavior differs from offline evaluation. Replacing files or models independently creates a period in which the system is internally inconsistent.
Index maintenance also consumes resources. Engines that store updates in immutable segments periodically merge those segments, and vector indexes may need additional structures built for the new data. On a write-heavy corpus, this work can reduce indexing throughput or delay when new documents become searchable even while query latency remains stable. Monitor indexing progress, merge activity, and index freshness alongside request performance.
The choices in this chapter set the evidence available to every later stage: analysis fixes which terms can match, chunking fixes which passages travel together, the embedding model fixes which semantic distinctions survive, and quantization fixes how much of that representation the first retrieval pass can use. Ranking and agentic orchestration can reorganize or request more of this evidence, but they cannot recover information the indexing pipeline discarded.
The next stage begins when a user supplies a query. Chapter 3 examines how the system interprets that incomplete request before retrieval begins.