# Setup and Corpus

Search Engineering · Implementation Guide · Chapter 1

Interactive edition: https://mutapa.dev/search-engineering/build/setup-and-corpus

Prepare a clean corpus and a reproducible Python environment for every chapter that follows.

## Outcome

A Python 3.14 environment and a local `corpus.jsonl` file containing one JSON document per line. Each document has a stable `doc_id`, a display `title`, and a searchable `body`. Later chapters build on this contract.

## The roadmap

This guide builds a retrieval-augmented system one runnable layer at a time. Chapter 2 establishes a BM25 keyword baseline. Chapter 3 embeds the documents, Chapter 4 searches those vectors, and Chapter 5 combines keyword and vector results with reciprocal rank fusion. Chapters 6 and 7 add generation and evaluation.

The worked example is a corpus of 284 films with hand-edited plot summaries and structured metadata. Familiar films make the first relevance checks quick, although useful evaluation still requires explicit judgments. You can substitute another corpus if it provides an id, a title, and searchable text.

## Prerequisites

- Python 3.14 (`python3.14 --version` verifies the interpreter).
- [`uv`](https://docs.astral.sh/uv/getting-started/installation/) for the locked environment.

## Setup

Create the environment from the checked-in lockfile:

```bash
uv sync --python 3.14 --locked
source .venv/bin/activate
```

The lockfile records the complete dependency graph tested by this guide. Its direct dependencies are NumPy, OpenAI, Pydantic, `rank-bm25`, `sentence-transformers`, and `faiss-cpu`, with exact versions in `pyproject.toml`.

Upgrade them deliberately with `uv lock --upgrade`, then rerun the guide before committing the new lockfile.

## Step 1: Download the source corpus

The chapter script downloads the source JSON from Cloudflare R2:

```text
https://data.mutapa.dev/corpora/films-284-v1.json
```

The versioned object is a JSON array of 284 films. Each film has stable identity, title, year, `directors`, origin countries, runtime, original languages, genres, a summary, external IDs, and provenance. The script verifies its SHA-256 digest before writing `corpus.json`, so a changed or incomplete download fails before conversion.

If you use another corpus, change `CORPUS_URL`, `SOURCE_SHA256`, and the field mapping in `convert()`.

## Step 2: Convert to `corpus.jsonl`

The remaining chapters require at least three fields: `doc_id`, `title`, and `body`. The converter maps each film summary to `body` and leaves unused metadata out of the retrieval contract.

```python
# code/chapters/chapter_00.py
import hashlib
import json
from pathlib import Path
from urllib.request import Request, urlopen

ROOT = Path(__file__).resolve().parents[2]
SOURCE = ROOT / "corpus.json"
OUT = ROOT / "corpus.jsonl"
CORPUS_URL = "https://data.mutapa.dev/corpora/films-284-v1.json"
SOURCE_SHA256 = "8c29ebf2dbeead70ac44c2e1ec430b841096f72698905349dd081c2426570008"

def download() -> None:
    request = Request(CORPUS_URL, headers={"User-Agent": "SearchEngineeringGuide/1.0"})
    with urlopen(request, timeout=30) as response:
        payload = response.read()

    digest = hashlib.sha256(payload).hexdigest()
    if digest != SOURCE_SHA256:
        raise ValueError(f"Corpus checksum mismatch: expected {SOURCE_SHA256}, got {digest}")

    SOURCE.write_bytes(payload)
    print(f"Downloaded {len(payload):,} bytes to {SOURCE.name}")

def convert() -> None:
    films = json.loads(SOURCE.read_text(encoding="utf-8"))
    with OUT.open("w", encoding="utf-8", newline="\n") as output:
        for film in films:
            record = {
                "doc_id": film["id"],
                "title": film["title"],
                "body": film["summary"],
            }
            output.write(json.dumps(record, ensure_ascii=False) + "\n")
    print(f"Wrote {len(films)} documents to {OUT.name}")
```

## Step 3: Validate the result

The final function parses every line and rejects missing or empty required fields. It then reports the corpus size, body-length range, and one complete record.

```python
def inspect() -> None:
    docs = []
    with OUT.open(encoding="utf-8") as corpus:
        for line_number, line in enumerate(corpus, start=1):
            document = json.loads(line)
            for field in ("doc_id", "title", "body"):
                if not isinstance(document.get(field), str) or not document[field].strip():
                    raise ValueError(f"Line {line_number} has an invalid {field}")
            docs.append(document)

    lengths = [len(document["body"].split()) for document in docs]
    print(f"{len(docs)} documents")
    print(
        f"body length (words): mean {sum(lengths) / len(lengths):.0f}, "
        f"min {min(lengths)}, max {max(lengths)}"
    )
    print("sample document:")
    print(json.dumps(docs[0], indent=2, ensure_ascii=False))

if __name__ == "__main__":
    download()
    convert()
    inspect()
```

## Run it

Run the script from the repository root:

```bash
python code/chapters/chapter_00.py
```

The output begins with:

```text
Downloaded 203,008 bytes to corpus.json
Wrote 284 documents to corpus.jsonl
284 documents
body length (words): mean 83, min 70, max 90
```

The script then prints the first complete document.

## Success criteria

- `python --version` reports Python 3.14.
- `uv sync --locked` completes without changing `uv.lock`.
- `python -c "import rank_bm25, numpy, openai"` succeeds.
- The script verifies the R2 object and writes 284 JSONL records, or the expected size of your replacement corpus.
- Every output record has a non-empty `doc_id`, `title`, and `body`.

## Troubleshooting

- **`URLError` or a timeout**: confirm that the R2 URL opens, then retry the download.
- **`Corpus checksum mismatch`**: remove any proxy or download step that modifies the response. If you intentionally replaced the corpus, calculate and set its new SHA-256 digest.
- **`JSONDecodeError`**: the downloaded object is not the expected JSON array. Verify `CORPUS_URL` before changing the converter.
- **Python 3.14 is unavailable**: run `uv python install 3.14`, then repeat `uv sync --python 3.14 --locked`.

## Production considerations

> **For production deployment**
>
> Real corpora need repeatable ingestion. Parse each source format into text, split long documents into retrieval-sized passages with parent references, and store a content hash so later runs process only changed documents. Preserve the stable-id, display-title, and searchable-text contract at the pipeline boundary.

## What you built

A `corpus.jsonl` file with one JSON document per line, each carrying a stable `doc_id`, a display `title`, and a searchable `body`. Every later chapter reads through this contract rather than the original source format.

The next chapter builds a BM25 keyword baseline over this corpus, producing the first retrieval results and the fixed queries that measure everything added afterward.

## Further reading

- **Diagnose corpus and indexing issues in an existing system:** [comprehensive runbook → Corpus & Indexing](https://mutapa.dev/search-engineering/reference/corpus-and-indexing.md)
- **Next chapter:** [Chapter 2, The Baseline: BM25](https://mutapa.dev/search-engineering/build/the-baseline-bm25.md)