# The Baseline: BM25

Search Engineering · Implementation Guide · Chapter 2

Interactive edition: https://mutapa.dev/search-engineering/build/the-baseline-bm25

Build an inspectable keyword baseline before adding semantic retrieval.

## Outcome

An in-memory BM25 scorer over the `corpus.jsonl` created in Chapter 1, plus a `search(query, k)` function that returns only matching documents and their scores.

Two fixed queries make the baseline inspectable. One shows where exact terms work. The other records a paraphrase failure that Chapter 4 must address.

## The role of the baseline

Keyword retrieval gives later changes a concrete comparison. Before adding embeddings, you need to see which results the corpus terms can recover and which requests use different language.

[BM25](https://doi.org/10.1561/1500000019) scores a document from the query terms it contains. It rewards discriminating terms, repeated evidence, and documents whose length remains useful for comparison.

For the diagnostic view of lexical retrieval, see [comprehensive runbook → Retrieval](https://mutapa.dev/search-engineering/reference/retrieval.md).

## Prerequisites

- Chapter 1 completed, with `corpus.jsonl` in the repository root.
- The locked Python 3.14 environment active. Run `uv sync --python 3.14 --locked` if you need to recreate it.

## Step 1: Load and tokenize the corpus

The loader resolves `corpus.jsonl` from the repository root and reads it as UTF-8. The type aliases separate source documents from results, which add a floating-point score.

```python
# code/chapters/chapter_01.py
import json
import re
from pathlib import Path

import numpy as np
from rank_bm25 import BM25Okapi

type Document = dict[str, str]
type SearchResult = dict[str, str | float]

ROOT = Path(__file__).resolve().parents[2]
CORPUS_PATH = ROOT / "corpus.jsonl"
TOKEN = re.compile(r"[a-z0-9']+")

def tokenize(text: str) -> list[str]:
    return TOKEN.findall(text.lower())

def load_corpus(path: Path = CORPUS_PATH) -> list[Document]:
    return [json.loads(line) for line in path.read_text(encoding="utf-8").splitlines()]
```

The tokenizer lowercases text and retains ASCII letters, digits, and apostrophes. Applying the same function to documents and queries keeps their token boundaries comparable.

This small tokenizer is intentionally limited. It does not normalize accented letters, inflections, or language-specific word boundaries. Change it only after a fixed query set shows which normalization the corpus needs.

## Step 2: Build the scorer

The scorer receives one token list per document. This implementation combines title and body because `rank-bm25` scores one text stream and does not assign separate field weights.

```python
DOCS = load_corpus()
BM25 = BM25Okapi(
    [tokenize(f"{document['title']} {document['body']}") for document in DOCS]
)
```

`BM25Okapi` uses `k1=1.5` and `b=0.75` by default. The `k1` parameter controls how quickly repeated occurrences stop adding much value. The `b` parameter controls how strongly document length changes the score.

*Inverse document frequency* gives more weight to query terms that occur in fewer documents. Together, these signals favor documents that contain distinctive query terms without rewarding repetition indefinitely.

This object is an in-memory scorer rather than an inverted index. Each query asks `rank-bm25` to score every document, which is acceptable for 284 films but does not describe a production retrieval architecture.

## Step 3: Search

The search function scores the query, removes documents with no positive match, and sorts the remaining indices by descending score. A stable sort keeps corpus order when scores tie.

```python
def search(query: str, k: int = 10) -> list[SearchResult]:
    if k < 1:
        return []

    scores = BM25.get_scores(tokenize(query))
    matching = np.flatnonzero(scores > 0)
    order = matching[np.argsort(scores[matching], descending=True, stable=True)[:k]]
    return [{**DOCS[index], "bm25_score": float(scores[index])} for index in order]
```

`descending=True` is the NumPy 2.5 sorting API. Filtering before the slice means `k` counts matching documents, while a query with no matching terms returns an empty list.

## Run it

Run the complete script from the repository root:

```bash
python code/chapters/chapter_01.py
```

For `locked jury room`, *12 Angry Men* ranks first because all three query terms appear in its title or summary. The terms provide direct lexical evidence for the result.

The second query, `space movie where the dad ages slowly`, does not retrieve *Interstellar* in the top five. Its summary expresses those ideas with different words, so BM25 has little useful evidence to score.

That failure is part of the baseline. Preserve both queries unchanged so Chapter 4 can compare semantic retrieval against the same requests.

## Success criteria

- `search("locked jury room", 5)` ranks *12 Angry Men* first.
- `search("qzxvplm", 5)` returns an empty list.
- Every returned result has a positive `bm25_score`.
- The two fixed queries preserve the observed exact-match success and paraphrase failure.

These checks establish coherent behavior rather than retrieval quality. Chapter 7 introduces judgments and aggregate metrics that can compare systems repeatably.

## Troubleshooting

- `FileNotFoundError`: run Chapter 1 from the repository root to create `corpus.jsonl`.
- `KeyError: 'body'`: confirm that each JSONL record follows the Chapter 1 contract.
- Empty results: print `tokenize(query)` and check whether any resulting token occurs in the corpus.
- Missing non-English terms: replace the ASCII tokenizer with one designed for the corpus language, then apply it to documents and queries.
- An unexpected top result: inspect its matched tokens before blaming BM25. The cause may be tokenization, combined fields, corpus text, or a semantic mismatch.

## Production considerations

> `rank-bm25` keeps the mechanism small enough to inspect, but it scans the corpus for every query. Production systems normally store postings in a persistent inverted index.
>
> Elasticsearch and OpenSearch provide analyzer pipelines, separate fields, field boosts, persistence, and distributed operations. SQLite FTS5 and Tantivy provide smaller embedded alternatives.
>
> Implementations also differ in defaults and score direction. Elasticsearch uses BM25 with `k1=1.2` and `b=0.75`. SQLite FTS5 uses the same constants but returns better matches as numerically lower values.
>
> Keep title and body separate in those systems so analyzers, field lengths, and weights remain independently configurable.

## What you built

An in-memory BM25 scorer over `corpus.jsonl` and a `search(query, k)` function that returns matching documents with their scores. Two fixed queries hold the baseline in place: one where exact terms succeed, and one paraphrase failure recorded for later chapters to answer.

The next chapter embeds the same documents, the first step toward recovering the paraphrase this baseline misses.

## Further reading

- [The Probabilistic Relevance Framework: BM25 and Beyond](https://doi.org/10.1561/1500000019)
- [Elasticsearch BM25 similarity](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules-similarity.html)
- [SQLite FTS5 BM25](https://www.sqlite.org/fts5.html#the_bm25_function)
- [Diagnose this stage in an existing system](https://mutapa.dev/search-engineering/reference/retrieval.md)
- [Next chapter: Chapter 3, Add Embeddings](https://mutapa.dev/search-engineering/build/add-embeddings.md)