Skip to content
Search Engineering

Implementation Guide · Chapter 2

The Baseline: BM25

Build an inspectable keyword baseline before adding semantic retrieval.

Outcome

An in-memory BM25 scorer over the corpus.jsonl created in Chapter 1, plus a search(query, k) function that returns only matching documents and their scores.

Two fixed queries make the baseline inspectable. One shows where exact terms work. The other records a paraphrase failure that Chapter 4 must address.

The role of the baseline

Keyword retrieval gives later changes a concrete comparison. Before adding embeddings, you need to see which results the corpus terms can recover and which requests use different language.

BM25 scores a document from the query terms it contains. It rewards discriminating terms, repeated evidence, and documents whose length remains useful for comparison.

For the diagnostic view of lexical retrieval, see comprehensive runbook → Retrieval.

Prerequisites

  • Chapter 1 completed, with corpus.jsonl in the repository root.
  • The locked Python 3.14 environment active. Run uv sync --python 3.14 --locked if you need to recreate it.

Step 1: Load and tokenize the corpus

The loader resolves corpus.jsonl from the repository root and reads it as UTF-8. The type aliases separate source documents from results, which add a floating-point score.

# code/chapters/chapter_01.py
import json
import re
from pathlib import Path

import numpy as np
from rank_bm25 import BM25Okapi

type Document = dict[str, str]
type SearchResult = dict[str, str | float]

ROOT = Path(__file__).resolve().parents[2]
CORPUS_PATH = ROOT / "corpus.jsonl"
TOKEN = re.compile(r"[a-z0-9']+")


def tokenize(text: str) -> list[str]:
    return TOKEN.findall(text.lower())


def load_corpus(path: Path = CORPUS_PATH) -> list[Document]:
    return [json.loads(line) for line in path.read_text(encoding="utf-8").splitlines()]

The tokenizer lowercases text and retains ASCII letters, digits, and apostrophes. Applying the same function to documents and queries keeps their token boundaries comparable.

This small tokenizer is intentionally limited. It does not normalize accented letters, inflections, or language-specific word boundaries. Change it only after a fixed query set shows which normalization the corpus needs.

Step 2: Build the scorer

The scorer receives one token list per document. This implementation combines title and body because rank-bm25 scores one text stream and does not assign separate field weights.

DOCS = load_corpus()
BM25 = BM25Okapi(
    [tokenize(f"{document['title']} {document['body']}") for document in DOCS]
)

BM25Okapi uses k1=1.5 and b=0.75 by default. The k1 parameter controls how quickly repeated occurrences stop adding much value. The b parameter controls how strongly document length changes the score.

Inverse document frequency gives more weight to query terms that occur in fewer documents. Together, these signals favor documents that contain distinctive query terms without rewarding repetition indefinitely.

This object is an in-memory scorer rather than an inverted index. Each query asks rank-bm25 to score every document, which is acceptable for 284 films but does not describe a production retrieval architecture.

The search function scores the query, removes documents with no positive match, and sorts the remaining indices by descending score. A stable sort keeps corpus order when scores tie.

def search(query: str, k: int = 10) -> list[SearchResult]:
    if k < 1:
        return []

    scores = BM25.get_scores(tokenize(query))
    matching = np.flatnonzero(scores > 0)
    order = matching[np.argsort(scores[matching], descending=True, stable=True)[:k]]
    return [{**DOCS[index], "bm25_score": float(scores[index])} for index in order]

descending=True is the NumPy 2.5 sorting API. Filtering before the slice means k counts matching documents, while a query with no matching terms returns an empty list.

Run it

Run the complete script from the repository root:

python code/chapters/chapter_01.py

For locked jury room, 12 Angry Men ranks first because all three query terms appear in its title or summary. The terms provide direct lexical evidence for the result.

The second query, space movie where the dad ages slowly, does not retrieve Interstellar in the top five. Its summary expresses those ideas with different words, so BM25 has little useful evidence to score.

That failure is part of the baseline. Preserve both queries unchanged so Chapter 4 can compare semantic retrieval against the same requests.

Success criteria

  • search("locked jury room", 5) ranks 12 Angry Men first.
  • search("qzxvplm", 5) returns an empty list.
  • Every returned result has a positive bm25_score.
  • The two fixed queries preserve the observed exact-match success and paraphrase failure.

These checks establish coherent behavior rather than retrieval quality. Chapter 7 introduces judgments and aggregate metrics that can compare systems repeatably.

Troubleshooting

  • FileNotFoundError: run Chapter 1 from the repository root to create corpus.jsonl.
  • KeyError: 'body': confirm that each JSONL record follows the Chapter 1 contract.
  • Empty results: print tokenize(query) and check whether any resulting token occurs in the corpus.
  • Missing non-English terms: replace the ASCII tokenizer with one designed for the corpus language, then apply it to documents and queries.
  • An unexpected top result: inspect its matched tokens before blaming BM25. The cause may be tokenization, combined fields, corpus text, or a semantic mismatch.

Production considerations

rank-bm25 keeps the mechanism small enough to inspect, but it scans the corpus for every query. Production systems normally store postings in a persistent inverted index.

Elasticsearch and OpenSearch provide analyzer pipelines, separate fields, field boosts, persistence, and distributed operations. SQLite FTS5 and Tantivy provide smaller embedded alternatives.

Implementations also differ in defaults and score direction. Elasticsearch uses BM25 with k1=1.2 and b=0.75. SQLite FTS5 uses the same constants but returns better matches as numerically lower values.

Keep title and body separate in those systems so analyzers, field lengths, and weights remain independently configurable.

What you built

An in-memory BM25 scorer over corpus.jsonl and a search(query, k) function that returns matching documents with their scores. Two fixed queries hold the baseline in place: one where exact terms succeed, and one paraphrase failure recorded for later chapters to answer.

The next chapter embeds the same documents, the first step toward recovering the paraphrase this baseline misses.

Further reading