Skip to content
Search Engineering

Implementation Guide · Chapter 1

Setup and Corpus

Prepare a clean corpus and a reproducible Python environment for every chapter that follows.

Outcome

A Python 3.14 environment and a local corpus.jsonl file containing one JSON document per line. Each document has a stable doc_id, a display title, and a searchable body. Later chapters build on this contract.

The roadmap

This guide builds a retrieval-augmented system one runnable layer at a time. Chapter 2 establishes a BM25 keyword baseline. Chapter 3 embeds the documents, Chapter 4 searches those vectors, and Chapter 5 combines keyword and vector results with reciprocal rank fusion. Chapters 6 and 7 add generation and evaluation.

The worked example is a corpus of 284 films with hand-edited plot summaries and structured metadata. Familiar films make the first relevance checks quick, although useful evaluation still requires explicit judgments. You can substitute another corpus if it provides an id, a title, and searchable text.

Prerequisites

  • Python 3.14 (python3.14 --version verifies the interpreter).
  • uv for the locked environment.

Setup

Create the environment from the checked-in lockfile:

uv sync --python 3.14 --locked
source .venv/bin/activate

The lockfile records the complete dependency graph tested by this guide. Its direct dependencies are NumPy, OpenAI, Pydantic, rank-bm25, sentence-transformers, and faiss-cpu, with exact versions in pyproject.toml.

Upgrade them deliberately with uv lock --upgrade, then rerun the guide before committing the new lockfile.

Step 1: Download the source corpus

The chapter script downloads the source JSON from Cloudflare R2:

https://data.mutapa.dev/corpora/films-284-v1.json

The versioned object is a JSON array of 284 films. Each film has stable identity, title, year, directors, origin countries, runtime, original languages, genres, a summary, external IDs, and provenance. The script verifies its SHA-256 digest before writing corpus.json, so a changed or incomplete download fails before conversion.

If you use another corpus, change CORPUS_URL, SOURCE_SHA256, and the field mapping in convert().

Step 2: Convert to corpus.jsonl

The remaining chapters require at least three fields: doc_id, title, and body. The converter maps each film summary to body and leaves unused metadata out of the retrieval contract.

# code/chapters/chapter_00.py
import hashlib
import json
from pathlib import Path
from urllib.request import Request, urlopen

ROOT = Path(__file__).resolve().parents[2]
SOURCE = ROOT / "corpus.json"
OUT = ROOT / "corpus.jsonl"
CORPUS_URL = "https://data.mutapa.dev/corpora/films-284-v1.json"
SOURCE_SHA256 = "8c29ebf2dbeead70ac44c2e1ec430b841096f72698905349dd081c2426570008"


def download() -> None:
    request = Request(CORPUS_URL, headers={"User-Agent": "SearchEngineeringGuide/1.0"})
    with urlopen(request, timeout=30) as response:
        payload = response.read()

    digest = hashlib.sha256(payload).hexdigest()
    if digest != SOURCE_SHA256:
        raise ValueError(f"Corpus checksum mismatch: expected {SOURCE_SHA256}, got {digest}")

    SOURCE.write_bytes(payload)
    print(f"Downloaded {len(payload):,} bytes to {SOURCE.name}")


def convert() -> None:
    films = json.loads(SOURCE.read_text(encoding="utf-8"))
    with OUT.open("w", encoding="utf-8", newline="\n") as output:
        for film in films:
            record = {
                "doc_id": film["id"],
                "title": film["title"],
                "body": film["summary"],
            }
            output.write(json.dumps(record, ensure_ascii=False) + "\n")
    print(f"Wrote {len(films)} documents to {OUT.name}")

Step 3: Validate the result

The final function parses every line and rejects missing or empty required fields. It then reports the corpus size, body-length range, and one complete record.

def inspect() -> None:
    docs = []
    with OUT.open(encoding="utf-8") as corpus:
        for line_number, line in enumerate(corpus, start=1):
            document = json.loads(line)
            for field in ("doc_id", "title", "body"):
                if not isinstance(document.get(field), str) or not document[field].strip():
                    raise ValueError(f"Line {line_number} has an invalid {field}")
            docs.append(document)

    lengths = [len(document["body"].split()) for document in docs]
    print(f"{len(docs)} documents")
    print(
        f"body length (words): mean {sum(lengths) / len(lengths):.0f}, "
        f"min {min(lengths)}, max {max(lengths)}"
    )
    print("sample document:")
    print(json.dumps(docs[0], indent=2, ensure_ascii=False))


if __name__ == "__main__":
    download()
    convert()
    inspect()

Run it

Run the script from the repository root:

python code/chapters/chapter_00.py

The output begins with:

Downloaded 203,008 bytes to corpus.json
Wrote 284 documents to corpus.jsonl
284 documents
body length (words): mean 83, min 70, max 90

The script then prints the first complete document.

Success criteria

  • python --version reports Python 3.14.
  • uv sync --locked completes without changing uv.lock.
  • python -c "import rank_bm25, numpy, openai" succeeds.
  • The script verifies the R2 object and writes 284 JSONL records, or the expected size of your replacement corpus.
  • Every output record has a non-empty doc_id, title, and body.

Troubleshooting

  • URLError or a timeout: confirm that the R2 URL opens, then retry the download.
  • Corpus checksum mismatch: remove any proxy or download step that modifies the response. If you intentionally replaced the corpus, calculate and set its new SHA-256 digest.
  • JSONDecodeError: the downloaded object is not the expected JSON array. Verify CORPUS_URL before changing the converter.
  • Python 3.14 is unavailable: run uv python install 3.14, then repeat uv sync --python 3.14 --locked.

Production considerations

For production deployment

Real corpora need repeatable ingestion. Parse each source format into text, split long documents into retrieval-sized passages with parent references, and store a content hash so later runs process only changed documents. Preserve the stable-id, display-title, and searchable-text contract at the pipeline boundary.

What you built

A corpus.jsonl file with one JSON document per line, each carrying a stable doc_id, a display title, and a searchable body. Every later chapter reads through this contract rather than the original source format.

The next chapter builds a BM25 keyword baseline over this corpus, producing the first retrieval results and the fixed queries that measure everything added afterward.

Further reading