Evaluation

Outcome

This chapter adds two offline evaluations. Retrieval qrels measure which labeled documents each ranker recovers and where they appear. Generation fixtures separate response status, citation provenance, and claim support.

Both inputs are checked into the repository. The runner validates them before calculating a score, so malformed judgments fail with a useful error instead of quietly changing the result.

Measurement before tuning

The previous chapters inspected a few searches directly. That works while a failure is obvious, but it cannot compare two systems across the same query distribution.

An evaluation set supplies that stable comparison. Its verdict is only as representative as its queries and judgments, so treat the set as a versioned model of the work users bring to the system.

The evaluation runbook covers sparse judgments, behavioral bias, and feedback failures in deployed systems.

Prerequisites

• Chapters 1 through 6 completed.
• The locked Python 3.14 environment active.
• corpus.jsonl and embeddings-local.npz built.

This run is offline. Select the local artifact explicitly because vector retrieval otherwise looks for the OpenAI artifact:

uv sync --python 3.14 --locked
source .venv/bin/activate
python code/chapters/chapter_00.py
python code/chapters/chapter_02_local.py
export EMBEDDINGS_FILE=embeddings-local.npz

Step 1: Define retrieval judgments

The repository includes qrels.jsonl. Each row has a stable query id, query text, and graded relevance judgments:

{"query_id":"q01","query":"space movie where the dad ages slowly","relevant":{"interstellar-2014":2}}
{"query_id":"q05","query":"boxer fights toward a championship","relevant":{"raging-bull-1980":2}}
{"query_id":"q17","query":"The Godfather","relevant":{"the-godfather-1972":2}}

Grade 2 means the summary directly satisfies the request. Grade 1 means it satisfies the subject and task but misses one stated constraint. Grade 0 means a reviewed candidate does not satisfy the request.

Absence does not prove irrelevance. These judgments cover known relevant films, not every film in the corpus. Recall therefore means recall over the labeled positives.

The set contains sixteen descriptive paraphrases and one exact-title query. That mix is intentional for this lesson, but it does not represent general traffic. Add observed query classes before using it for a product decision.

Step 2: Validate the inputs

load_qrels() rejects blank lines, malformed JSON, duplicate query ids, empty queries, invalid grades, missing positive judgments, and document ids absent from the corpus.

def load_qrels(path: Path, corpus_path: Path = CORPUS_PATH) -> list[dict]:
    rows = _read_jsonl(path)
    corpus_ids = {
        json.loads(line)["doc_id"]
        for line in corpus_path.read_text(encoding="utf-8").splitlines()
    }
    seen_query_ids: set[str] = set()
    for index, row in enumerate(rows, start=1):
        query_id = row.get("query_id")
        query = row.get("query")
        relevant = row.get("relevant")
        if not isinstance(query_id, str) or not query_id.strip():
            raise ValueError(f"qrels row {index} needs a query_id")
        if query_id in seen_query_ids:
            raise ValueError(f"duplicate query_id: {query_id}")
        seen_query_ids.add(query_id)
        if not isinstance(query, str) or not query.strip():
            raise ValueError(f"qrels row {index} needs a query")
        if not isinstance(relevant, dict) or not relevant:
            raise ValueError(f"qrels row {index} needs relevance judgments")
        unknown = sorted(set(relevant) - corpus_ids)
        if unknown:
            raise ValueError(f"qrels row {index} has unknown ids: {', '.join(unknown)}")
    return rows

The runnable file also checks that every grade is a nonnegative integer and every query has at least one positive judgment.

Step 3: Calculate retrieval metrics

Recall@k asks how many labeled positives enter the top k. Reciprocal rank measures the position of the first positive. Mean reciprocal rank, or MRR, averages that value across queries.

Normalized Discounted Cumulative Gain, or nDCG, uses the grades and discounts lower positions. This implementation uses linear gain, so grade 2 contributes twice the undiscounted gain of grade 1.

def recall_at_k(ranked_ids: list[str], relevant: dict[str, int], k: int) -> float:
    _validate_ranking(ranked_ids, k)
    targets = _positive_ids(relevant)
    return len(set(ranked_ids[:k]) & targets) / len(targets)

def reciprocal_rank(ranked_ids: list[str], relevant: dict[str, int]) -> float:
    _validate_ranking(ranked_ids, max(1, len(ranked_ids)))
    targets = _positive_ids(relevant)
    return next(
        (1.0 / rank for rank, doc_id in enumerate(ranked_ids, start=1)
         if doc_id in targets),
        0.0,
    )

def ndcg_at_k(ranked_ids: list[str], relevant: dict[str, int], k: int) -> float:
    _validate_ranking(ranked_ids, k)
    _positive_ids(relevant)
    dcg = sum(
        relevant.get(doc_id, 0) / math.log2(rank + 1)
        for rank, doc_id in enumerate(ranked_ids[:k], start=1)
    )
    ideal = sum(
        grade / math.log2(rank + 1)
        for rank, grade in enumerate(sorted(relevant.values(), reverse=True)[:k], start=1)
    )
    return dcg / ideal

Duplicate document ids invalidate a ranking. Counting duplicates as separate hits could otherwise push recall above 1. Invalid k values and judgment sets without a positive document also fail before arithmetic begins.

Step 4: Compare retrievers

evaluate() keeps the ranking and metrics for every query. The aggregate describes the set; the rows explain it.

def evaluate(search_fn: SearchFunction, qrels: list[dict], k: int = 10) -> dict:
    if not qrels:
        raise ValueError("qrels must not be empty")
    if k < 1:
        raise ValueError("k must be positive")
    per_query = []
    for entry in qrels:
        results = search_fn(entry["query"], k)
        ranked_ids = [str(result["doc_id"]) for result in results]
        _validate_ranking(ranked_ids, k)
        per_query.append({
            "query_id": entry["query_id"],
            "query": entry["query"],
            "ranked_ids": ranked_ids,
            "recall": recall_at_k(ranked_ids, entry["relevant"], k),
            "rr": reciprocal_rank(ranked_ids, entry["relevant"]),
            "ndcg": ndcg_at_k(ranked_ids, entry["relevant"], k),
        })
    count = len(per_query)
    return {
        "recall@k": sum(row["recall"] for row in per_query) / count,
        "mrr": sum(row["rr"] for row in per_query) / count,
        "ndcg@k": sum(row["ndcg"] for row in per_query) / count,
        "per_query": per_query,
    }

Run the three systems with the same corpus, artifact, queries, and cutoff:

EMBEDDINGS_FILE=embeddings-local.npz python code/chapters/chapter_06.py

The locked environment produces:

system    recall@10     mrr  ndcg@10
bm25          0.716   0.718    0.695
vector        0.931   0.884    0.886
hybrid        0.892   0.764    0.783

Vector retrieval wins on this set because the set is dominated by paraphrases. That result supports a statement about these judgments rather than a universal preference for vector retrieval.

For the Interstellar query, vector retrieval ranks the target first and BM25 misses it. RRF combines those lists and places it ninth. The per-query row exposes the loss hidden by the average.

Add exact titles, identifiers, misspellings, and other observed query classes before choosing a production retriever. Report scores by query class as well as across the full set.

Step 5: Evaluate grounded generation

Retrieval metrics cannot tell whether a generated claim follows from its citation. generation_eval.jsonl supplies four reviewed response fixtures for the contract built in Chapter 6.

Each fixture records the expected response status, retrieved ids, structured response, and a human support judgment for every claim-citation pair.

def evaluate_generation(path: Path = GENERATION_CASES_PATH) -> dict:
    cases = _read_jsonl(path)
    status_hits = provenance_hits = supported_claims = claim_count = 0
    for index, case in enumerate(cases, start=1):
        response = GroundedResponse.model_validate(case.get("response"))
        retrieved_ids = set(case.get("retrieved_ids", []))
        support = case.get("support", {})
        status_hits += response.status == case.get("expected_status")
        provenance_hits += check_response(response, retrieved_ids)["passed"]
        for claim_index, claim in enumerate(response.claims):
            judgments = support.get(str(claim_index))
            if not isinstance(judgments, dict) or set(judgments) != set(claim.citations):
                raise ValueError(f"generation case {index} lacks complete support judgments")
            claim_count += 1
            supported_claims += any(judgments.values())
    return {
        "status_accuracy": status_hits / len(cases),
        "provenance_pass_rate": provenance_hits / len(cases),
        "claim_support_rate": supported_claims / claim_count if claim_count else 1.0,
    }

The runnable file also validates case ids, retrieved-id lists, and boolean support judgments before scoring.

The sample deliberately includes a valid citation attached to an unsupported claim. Provenance passes that case; claim support does not. Another case refuses despite sufficient evidence, lowering status accuracy.

The fixture report is:

{'status_accuracy': 0.75, 'provenance_pass_rate': 1.0, 'claim_support_rate': 0.5}

These fixtures test evaluation logic without a model call. For a live system, save structured responses, remove sensitive text, and have reviewers judge a sampled set against the cited passages.

Step 6: Compare system prompts and model candidates

The fixture evaluator proves that the scoring logic works on reviewed responses. A release decision also needs to run the generator itself over fixed evidence. The repository includes a Promptfoo configuration in evals/promptfoo/ for that comparison.

The matrix contains two system prompts, a small-model candidate, a large-model candidate, and three cases: supported evidence, insufficient evidence, and an instruction embedded in a retrieved document. The Python provider calls the same GroundedResponse contract used by Chapter 6 and returns structured output, token usage, and latency to the framework.

Keep concrete model ids in the environment rather than the configuration so the route categories remain stable when approved models change:

export OPENAI_API_KEY=...
export SMALL_MODEL=...
export LARGE_MODEL=...
export SMALL_INPUT_COST_PER_MILLION=...
export SMALL_OUTPUT_COST_PER_MILLION=...
export LARGE_INPUT_COST_PER_MILLION=...
export LARGE_OUTPUT_COST_PER_MILLION=...
bun run eval:prompts

Use the current provider prices in US dollars per million tokens. Keeping rates beside model ids makes the reported cost explicit and reviewable when prices change.

Use bun run eval:prompts:view to inspect the per-case matrix. The assertions check response state, citations, refusal shape, and resistance to a retrieved instruction. Add human support judgments for semantic claims before treating the suite as a release gate.

Hold the question and context fixed when comparing prompts. A retrieval change belongs in the retrieval report; changing retrieval and prompt together makes an improvement impossible to attribute. Preserve the generated result artifact with the prompt files, model ids, rubric, and date. Latency and token totals are comparison dimensions, while quality remains the gate.

Interpret failures by layer

Low retrieval recall means the labeled evidence did not enter the context window. Low reciprocal rank with healthy recall means the evidence was found but buried.

Low status accuracy means the model answered when it should have declined, or declined despite sufficient evidence. Low provenance means the structured response violated its citation contract.

Low claim support means cited documents do not justify the claims attached to them. Prompt changes cannot recover evidence that retrieval never supplied, so keep retrieval and generation reports separate.

Judgment sources

Start with reviewed labels from people who understand the corpus and task. Record the rubric, assessor, date, and disagreements. Increase the set until important query classes have enough examples to inspect separately.

Behavioral events add scale after deployment, but clicks are observations rather than direct relevance labels. Position and presentation affect what users can examine and choose.

Log a unique search-instance id, the ordered result list, and subsequent events. The OpenSearch UBI schema is one current implementation of that linked event model.

Models can propose additional judgments, but their agreement with your rubric must be measured on a held-out human-labeled set. Keep model-generated labels identified by source and subject them to review.

Success criteria

• The checked-in qrels validate against the current corpus.
• Every metric for valid input remains between 0 and 1.
• Duplicate rankings, unknown ids, empty qrels, invalid grades, and invalid k values fail clearly.
• Retrieval and generation reports remain separate.
• Every generation claim has a complete human support judgment.
• System-prompt and model candidates run over the same fixed evidence and per-case assertions.
• A system comparison includes aggregate, per-query, and query-class inspection.

Production considerations

At larger scale, use a maintained evaluation library and verify it in the same Python environment as the pipeline. ranx 0.3.21 installs on Python 3.14 and provides ranking metrics plus system comparison.

BEIR 2.2.0 supplies heterogeneous retrieval benchmarks. It is useful for broad comparison, but domain qrels remain the release gate for a domain system.

Report uncertainty and per-query differences before declaring a winner. A mean without the paired rows can conceal that a change helps one query class and harms another.

Offline evaluation gates candidates. Online experiments then measure user outcomes under controlled exposure. Do not train on behavioral labels until their collection and bias assumptions are documented.

What you built

The guide now has a repeatable evidence loop. Retrieval evaluation measures whether labeled sources arrive and where they rank. Generation evaluation measures whether the response chooses the right state and uses those sources faithfully.

The numbers remain conditional on the judgment set. Chapter 8 uses these failure signatures to choose the next change instead of selecting an upgrade because it sounds promising.

Further reading

• Evaluation and feedback runbook
• ranx 0.3.21
• OpenSearch User Behavior Insights
• Next chapter: Chapter 8, Where to Go Next