Every instrument in this course searches the same corpus: 250 films, each represented by a title, release year, directors, origin countries, runtime, original language, genre labels, and a plot summary. Chapter 1 used those records to judge whether a result could satisfy a request. Before any request arrives, however, the search system must turn the records into an artifact that can find supporting evidence without scanning and interpreting every film from the beginning.
That preparation happens offline, when the exact language of future queries is still unknown. The team must decide which written forms count as a match, what portion of a document retrieval will score, how semantic similarity will be represented, and how much numerical detail the index can afford to keep. Each decision makes the online path faster by settling work in advance, but it also sets a limit. If the prepared artifact removes a distinction or separates facts that belong together, no ranking model or agent can restore that evidence when the query arrives. The first decision is what the index will recognize as the same searchable term.
Decide which terms can match
A lexical search system works from stored terms. Before its index can store a document, the system runs the text through an analysis chain, an ordered sequence of transformations whose output becomes the document’s searchable form. The chain may split text on spaces or punctuation, normalize letter case, remove common words, and reduce related word forms to a shared base. The resulting units are called tokens, and only compatible document and query tokens can produce an exact match.
The instrument below applies a small analysis chain to an editable document and the fixed query “x wing engine.” Both pass through the same steps. Toggle one step at a time and compare the resulting tokens, then watch which query tokens gain or lose an exact match.
At first, the query finds nothing in the sentence, even though the connection is obvious to a reader. Lowercasing removes the difference between X and x, while splitting on punctuation turns “X-Wing” into x and wing. Stemming then reduces engines to engine, supplying the third match. Removing stopwords does not affect this query, but it still changes which words the index stores.
The three matches still depend on the user writing x wing as two words. A query for xwing would remain different from the indexed tokens x and wing, so supporting both forms requires another normalization rule or an explicit domain mapping. Analysis can preserve known variations in a shared form, but it cannot anticipate every way users may express the same concept.
The engine match also shows the tradeoff introduced by stemming. Removing endings by rule can improve recall, the system’s ability to retrieve relevant results, because related forms such as engine and engines produce the same token. A crude stemmer can also merge terms that should remain distinct: a suffix rule that reduces both universal and university to univers makes them indistinguishable in the index. Lemmatization uses a language’s vocabulary and morphology to map a word to its dictionary form. It makes a more informed choice than suffix stripping, but it requires language-specific resources. Either approach determines which variations match and which distinctions the index preserves.
Production systems often configure separate analysis chains for documents and queries. The chains do not have to be identical: a query analyzer might expand a known synonym while the document index stores only the preferred term. They do have to produce compatible tokens. If the document chain stores x and wing while the query chain produces xwing, the lexical scorer receives no shared term to score. When an exact term or phrase cannot be retrieved, inspect the output on both sides before changing the ranking model. Those compatible token streams are the first evidence the index must store.
▸Implement it: tokenize the course corpuscode/chapters/chapter_01.py
def tokenize(text: str) -> list[str]:
return TOKEN.findall(text.lower())Store enough evidence to score
Once analysis has produced tokens, a lexical search system records them in an inverted index. For each token, the index stores a list of the documents that contain it. This reverses the corpus: instead of scanning every document for a query term, the engine can move directly from that term to a much smaller set of candidates.
The index also preserves evidence needed to score those candidates. It records how often a term appears in each document, the length of each document, and corpus statistics such as average document length. These values become inputs to BM25, a widely used keyword ranking function. Chapter 4 explains how BM25 combines them for the current query. Much of what looks like query-time scoring therefore depends on measurements prepared offline.
Meaning-based retrieval prepares a different comparison for the same candidates. An embedding model converts each candidate into a vector, a fixed-length list of numbers that represents features of its meaning. In a bi-encoder, the model encodes documents before search and encodes the query separately when it arrives. Retrieval then treats the distance between the query vector and stored document vectors as evidence of semantic similarity.
The comparison can use only distinctions the embedding model preserved. Model selection is therefore an indexing decision, not merely a query-time choice. Public benchmarks can narrow the options by language support, model size, license, and input length, but the final model must be tested against the team’s own corpus and relevance judgments. Chapter 7 develops that evaluation process. Once selected, the model must encode both sides of the comparison: a new or fine-tuned query encoder cannot be paired reliably with document vectors from the old model.
Lexical postings and document vectors still need an object to describe. A retrieval unit is the record the system scores and returns as a candidate, so the next decision is how much source material that object should contain.
Decide what one result contains
Each film summary in this course is about eighty words and already works as one retrieval unit. Real documents rarely arrive in such convenient pieces. A long manual may contain many unrelated procedures, and returning the entire manual gives the reader too much material to inspect. Search systems therefore divide long documents into smaller units called chunks before indexing them.
Chunk size determines how much evidence travels together in one result. A small chunk can isolate the passage that closely matches a query, but it may omit definitions or qualifications from nearby text. A large chunk preserves more context, but unrelated material can weaken the retrieval signal and consume more of the system’s processing budget. The boundary between chunks introduces another risk: information needed for one answer may be divided across two candidates.
The course’s short film summaries do not make boundary loss easy to see, so the instrument below uses a longer bicycle review with a known answer span. Each panel shows a retrievable chunk containing at least part of the underlined answer. At the default settings, the recommendation is divided between two panels. Change the size, overlap, or split strategy and watch whether one candidate can carry the complete answer.
QuestionWhat tire pressure does the reviewer recommend for cold weather?
I bought the Meridian Alloy last spring for long winter rides on the coast road, and after four months of rough pavement it has quietly become my default training bike. For cold-weather rides I drop to 82 psi in the
rear and 78 up front, which keeps the ride compliant without risking pinch flats. At summer pressures the same wheels feel harsh once the temperature falls, so that adjustment matters more than any single component choice. The rest of the
Increasing the chunk size creates more room for the answer, while overlap repeats the end of one chunk at the beginning of the next so the passage gets another chance to remain intact. Cutting every n words is fixed-size chunking; adding overlap turns it into a sliding window, which spends additional index space as insurance against a bad boundary. Both strategies remain blind to the document, which is why the first boundary can land inside the recommendation.
Structure-aware strategies use the document to avoid some of that duplication. A recursive splitter tries paragraph breaks first, then sentence breaks, then word boundaries, falling back only when a piece remains too large. A document-aware pipeline chooses the preferred separators from the format, such as Markdown headings, HTML sections, or the page and section structure of a manual. Switch the instrument to sentence boundaries and the complete recommendation remains intact at every size setting because it occupies one sentence.
Hierarchical chunking separates the unit that matches from the unit the reader receives. The system indexes small child chunks so retrieval can match precisely, then returns each child’s larger parent section so generation receives the surrounding definitions and qualifications. The price is a second layer of storage and a parent mapping that must survive index rebuilds.
Neither preserving a whole sentence nor returning a parent section guarantees that retrieval will find the needed passage. The right strategy depends on the documents and requests the system must handle. Build a small evaluation set with representative questions and known answer passages, then measure whether the chunker keeps those passages intact and whether retrieval returns them. Compare those results with the storage, latency, and context costs of each configuration. Chunk size and overlap then become testable indexing choices rather than fixed defaults.
Late chunking addresses a related loss in embedding-based retrieval. Instead of splitting the document first and embedding each chunk without its surroundings, the model processes the longer document at the token level before grouping those token representations into chunk vectors. The resulting vectors can preserve context from outside a chunk, such as the subject of a pronoun or a definition from an earlier paragraph. The method requires suitable token-level model output and enough input capacity for the source document, so it should be evaluated against conventional chunking on the same requests.
▸Implement it: split documents with overlapping windowscode/chunk.py
def chunk(words: list[str], size: int = 40, overlap: int = 10) -> list[list[str]]:
"""Fixed-size sliding window with overlap. Overlap is cheap insurance:
a boundary that lands inside an answer span destroys it for retrieval,
and overlapping windows give every span a second chance to sit whole
inside some chunk."""
step = max(size - overlap, 1)
return [words[i : i + size] for i in range(0, len(words), step)]Choose how much vector detail to keep
The completed chunks give the vector index many more representations to store. The 384-dimensional vectors for this course’s 250 films require less than half a megabyte when each value uses the 32-bit float32 format. Across millions of chunks, however, those values consume substantial storage and memory bandwidth. The team must decide how much numerical precision broad retrieval needs and which higher-precision representation later scoring should retain.
Quantization reduces the number of bits used for each vector value. Converting float32 values to 16-bit bfloat16 halves their size, while scalar int8 quantization stores one 8-bit integer per dimension and reduces vector storage by 75 percent. Binary quantization keeps one bit per dimension, reducing storage by about 97 percent, and commonly compares those vectors with Hamming distance, the number of bit positions at which they differ. Each step makes the first-stage index smaller and cheaper to search by removing more numerical detail.
Measure that loss by comparing retrieval from the compressed index with retrieval from the full-precision vectors. For representative queries, record how many full-precision nearest neighbors remain at a chosen cutoff, such as the first ten results, then repeat the comparison with a larger compressed candidate set. This measures fidelity to the original vector search, which differs from recall against human relevance judgments. A compressed index can disagree with the full-precision ranking and still return useful results, while perfect agreement can preserve the mistakes of a poor embedding model. Both tests matter because compression quality depends on the model, number of dimensions, and corpus being searched.
A two-stage search can recover much of the lost ordering accuracy. First, retrieve more candidates than the user will see from the compressed index. Then rescore that larger set with higher-precision vectors and keep the best results. The compressed representation supports broad, inexpensive retrieval, while the more precise representation makes the final distinctions. Chapter 5 applies this pattern to ranking and shows why the candidate window must be measured rather than assumed.
This recovery pattern requires the higher-precision vectors to remain available. They can live on disk rather than in the memory used by the first-stage index, but rescoring still needs to read them. If the pipeline discards them, later stages cannot restore the missing precision without encoding the corpus again. The compressed and precise copies therefore belong to one compatible index release.
Release the corpus and index as one compatible artifact
Source documents can change before the indexing pipeline does. A manual may describe several hardware revisions, and a film may have a theatrical cut and a director’s cut. Keep one stable identity for the work or document family, give each revision an immutable version identity, and attach that identity and its applicability metadata to every chunk. A remake is a separate work connected by lineage rather than another version of the same film. These distinctions let retrieval select the right revision without silently mixing evidence from related but incompatible sources.
Those source versions feed a particular preparation pipeline. The same corpus snapshot will produce a different searchable artifact when its schema, analysis chain, chunking rules, embedding model, or quantization settings change. A production release should identify each of those inputs so the result can be reproduced. Its query analyzer must produce tokens compatible with the lexical index, its query encoder must match the stored document vectors, and its rescoring stage must read the higher-precision vectors built for the same compressed index. Mixing versions can degrade retrieval without causing a request error.
Deploy these parts as one coordinated release. Build the new version alongside the artifact currently serving traffic, verify its document counts and a representative set of queries, then switch the online path to the complete release. Keeping the previous version available makes rollback possible if production behavior differs from offline evaluation. Replacing files or models independently creates a period in which the system is internally inconsistent.
Maintaining that release also consumes resources. Some engines write updates into read-only segments and periodically merge them, while vector indexes may need additional search structures built for new data. On a write-heavy corpus, this work can reduce indexing throughput or delay when documents become searchable even while query latency remains stable. Monitor indexing progress, merge activity, and freshness alongside request performance.
The offline path has now produced a versioned artifact with compatible tokens, retrieval units, vectors, and source identities. That artifact fixes the language and evidence the online system can use. When a user supplies a short request, Chapter 3 examines how query understanding can express its incomplete meaning in terms this index can retrieve.