Implementation Guide · Chapter 8

Where to Go Next

Choose the next change from measured failures, route model work by difficulty, and run an observable bounded search-revision loop.

Outcome

This chapter closes the core build with three artifacts: a measured upgrade proposal, an inspectable model router, and a bounded agentic_answer() loop with an operational trace. The proposal controls what should change. The router allocates model capacity. The loop shows when another search is justified and when execution must stop. The governed continuation then adds permission-aware retrieval, policy-scoped state, and connector contracts.

The baseline you now have

The preceding seven chapters built a corpus contract, lexical and vector retrieval, fusion, grounded generation, and separate retrieval and generation evaluations.

The embedding artifacts contain corpus checksums and model metadata. They do not cache embedding work. The system is a command-line baseline over one small corpus rather than a serving layer.

Its constraints are now observable. That makes each limitation a candidate for measured work rather than a reason to replace the whole system.

Write the upgrade proposal first

Record the decision before adding machinery. A useful proposal contains these fields:

Observed symptom:
Affected query class:
Baseline metric and sample size:
Proposed change:
Latency and cost budget:
Acceptance threshold:
Failure and safety checks:
Rollback condition:

The symptom should identify a layer without assuming a cause. Low paraphrase recall can begin in corpus coverage, labels, query interpretation, chunking, or embeddings. Inspect failed rows before choosing one.

Chapter 7 provides retrieval, response-status, provenance, and claim-support measurements. Preserve that separation in the proposal so a gain in one layer cannot hide a loss in another.

Choose the smallest justified change

A batch-built index remains appropriate when freshness and update latency meet the requirement. Add incremental indexing when stale documents become a measured failure.

Apply tenant and authorization filters before candidate retrieval. Location, availability, and other request context may also belong in pre-filtering when documents outside that scope must not enter the candidate set.

Add a request-time loop only when another bounded search can recover a class of failures. A loop adds model calls, latency, new failure states, and a larger evaluation surface.

If ranking wastes good candidates

Healthy recall with weak MRR or nDCG means labeled documents enter the candidate window but rank too low. Confirm that pattern across a query class before adding a second pass.

A cross-encoder reads the query and document together, then scores a limited candidate set. Candidate depth is a tuned latency and quality parameter rather than a fixed value such as 50.

Register the reranker explicitly in Chapter 7’s systems dictionary. The existing compare() function does not discover new retrievers automatically.

Learning to rank requires more data than the demonstration qrels provide. Build separate training and evaluation sets, preserve impression-time feature values, and prevent query or document leakage between them.

See Course Chapter 5, Ranking and Fusion and the ranking runbook.

If the request is misinterpreted

Query understanding is implicated when spelling, domain terms, ambiguity, or structured constraints prevent retrieval from seeing the intended request.

The current corpus cannot filter on genre, year, or director because those fields were discarded during conversion. Add fields to the corpus contract, index them, and rebuild before extracting those filters from a query.

Cache an interpretation only when reuse is safe. Include the model, prompt, taxonomy, locale, tenant, authorization scope, and relevant schema version in the key. Invalidate entries when any input changes.

Measure rewrites against the original query. A rewrite that raises retrieval scores while changing intent is a regression.

See Course Chapter 3, Query Understanding and the query-understanding runbook.

If vector search exceeds its budget

At 284 documents, exact float32 search is inexpensive in the tested environment. At larger scale, measure index memory, query latency, update cost, and relevance before choosing compression or approximation.

Moving from 32-bit floats to 8-bit values reduces raw vector width by 75 percent. One-bit values reduce it by 96.875 percent. Index overhead means total storage will fall by a different amount.

Recall loss is specific to the model, corpus, index, and query distribution. Select over-fetch depth empirically and rerank candidates against retained full-precision vectors.

Chapter 5’s overfetch widens the BM25 and vector lists before RRF. It is not a quantized-index recovery pass, and this implementation guide has not yet built one.

Measure quantized recall against exact full-precision results. Keep that internal approximation metric separate from Chapter 7’s relevance recall over human judgments.

See Course Chapter 2, Corpus and Indexing, Course Chapter 4, Retrieval, and the retrieval runbook.

Route model work by difficulty

route_model() applies a deterministic policy to task type, output constraints, ambiguity, tool requirements, and input size. The result is an inspectable RouteDecision, not only a model id:

class RouteDecision(BaseModel):
    route: Literal["small", "large"]
    model_id: str
    reason: str
    policy_version: str
    prompt_version: str
    token_budget: int
    latency_budget_ms: int
    fallback_model_id: str | None

Constrained classification, extraction, judging, and grounded generation can enter the small route when their input fits its budget. Planning, ambiguity, tool use, open-ended output, and oversized input enter the large route. Keep the concrete model ids in configuration:

policy = RoutingPolicy(
    small_model="small-candidate",
    large_model="large-candidate",
    max_small_input_tokens=4_000,
)
decision = route_model(request, policy)

routed_generate() looks up the selected adapter, calls it, and attaches the complete route decision to the result. The fallback id is a policy declaration rather than an automatic retry. The caller should escalate only on a typed failure such as schema validation, unsupported tool use, or a declared confidence threshold, then record both attempts.

Evaluate the small and large candidates with Chapter 7’s fixed prompt cases before changing the policy. Compare quality, status and citation validity, fallback rate, latency, tokens, and cost by task class. A router call made by another model belongs only when its measured gain pays for that extra call.

Build a bounded search-revision loop

The runnable loop lives in code/chapters/chapter_07.py. It accepts three capabilities rather than creating provider clients internally:

search(query, k) returns an authorized ranked list.
assess(question, query, results) returns a validated Assessment.
generate(question, results) creates a grounded response from the accepted evidence.

In the vocabulary of the course and runbook, assess plays the role of the judge inside the loop, and the code around agentic_answer acts as the harness. Dependency injection keeps the control policy testable. A production adapter owns its provider timeout, retry policy, credentials, rate limit, and tenant scope.

If assessment uses a model, treat result text as untrusted data and request a structured response matching Assessment. The loop validates that response before reading its fields.

class Assessment(BaseModel):
    score: float = Field(ge=0.0, le=1.0)
    sufficient: bool
    rewrite: str = ""
    reason: str = ""

The score ranks attempts. sufficient records the assessor’s decision. Acceptance requires both, preventing a contradictory low score from ending the loop.

The complete function validates the question, budgets, acceptance threshold, result shape, assessment, and generation status. It never calls generation with an empty result list. Every terminal path runs through one local finish() function, which attaches the same request-level trace whether the loop answers, refuses, repeats a query, exhausts its budget, or encounters an error. The loop measures search, assessment, and generation separately on each attempt.

Repeated rewrites stop with repeated_query. Missing rewrites stop with insufficient_context. The attempt budget stops unresolved work with budget_exhausted.

The best attempt is retained for diagnosis, but weak evidence is not sent to generation merely because the budget ended.

Chapter 6 now exposes answer_from_results(question, results), which can serve as the generation adapter. The search adapter should close over the caller’s authorization scope.

Evaluate the trajectory

A better final answer does not excuse unbounded work. The returned trace records request id, terminal status, total latency, attempt count, model-route decisions, input and output tokens, and typed errors. Each attempt also retains stage timings and result ids.

def trajectory_metrics(run: dict) -> dict:
    scored = [attempt["score"] for attempt in run["attempts"] if "score" in attempt]
    trace = run.get("trace", {})
    usage = trace.get("usage", {})
    return {
        "status": run["status"],
        "search_calls": len(run["attempts"]),
        "first_score": scored[0] if scored else None,
        "final_score": scored[-1] if scored else None,
        "score_improvement": scored[-1] - scored[0] if scored else None,
        "answered": run["status"] == "answered",
        "total_latency_ms": trace.get("total_ms"),
        "model_ids": [route["model_id"] for route in trace.get("model_routes", [])],
        "input_tokens": usage.get("input_tokens", 0),
        "output_tokens": usage.get("output_tokens", 0),
        "errors": trace.get("errors", []),
    }

Compare the loop with the one-pass baseline on the same cases. Require a declared quality improvement within latency and cost budgets. Evaluate the external outcome rather than accepting the loop’s own score as proof.

In production, add queue time and time to first token when output is streamed. Preserve fan-out and join boundaries if tools run concurrently so the trace exposes the critical path. Aggregate the median (p50) and tail percentiles by route, model, prompt version, index version, cache state, attempt count, and terminal status. Raw query and document text need separate privacy, access, sampling, and retention rules from categorical trace fields.

Run the deterministic example without an API key:

python code/chapters/chapter_07.py

The example misses Interstellar, rewrites once, finds it, and stops after two searches. The checked-in tests cover the public routing decisions, real dispatch, and the operational trace contract. Extend them with repeated rewrites, empty results, invalid assessments, tool errors, generation errors, and budget exhaustion as those adapters become production code.

See Course Chapter 8, Agentic Orchestration and the agentic-orchestration runbook.

Migrate only for measured operational needs

Concurrent traffic, incremental updates, replication, filtering, consistency, backup and restore, and observability can justify moving beyond scripts. Write the required behavior and failure budget before selecting an engine.

OpenSearch and Elasticsearch provide lexical, vector, and rank-fusion features through different APIs. PostgreSQL provides full-text ranking, while pgvector adds vector search; their hybrid RRF is assembled in application SQL.

Use an adapter that preserves (query, k) -> ranked records so Chapter 7 can evaluate the candidate engine. Set release thresholds for quality, latency, update behavior, and failures rather than requiring identical rankings.

Promote compatible runtime artifacts

Version the document schema, analyzer, query transform, embedding model, vector dimension, index, reranker, and generation contract. Route each request to a compatible set.

Version evaluation sets with experiment results, but do not deploy them as runtime dependencies. Preserve the exact set used to approve a release so later comparisons remain interpretable.

Build a candidate set offline, run retrieval and generation evaluations, then canary traffic with explicit success and rollback thresholds. Keep the previous complete runtime set recoverable.

Ranking training rows must retain impression-time feature values. Joining current inventory, price, or popularity to an old interaction leaks future information into training.

Success criteria

The upgrade proposal names evidence, budget, threshold, and rollback condition.
The loop stops on acceptance, missing rewrite, repeated query, tool failure, generation failure, or budget exhaustion.
Only permission-scoped search results reach assessment and generation.
Trajectory evaluation compares against the one-pass baseline.
A migration candidate passes declared quality and operational thresholds.
Runtime artifacts can be rolled back as one compatible set.

The next measured step

Run Chapter 7’s worst-query report and group failures by query class. Choose one class, write the upgrade proposal, and add its acceptance test before implementing the change.

That cycle is the production path: preserve the evidence boundary, change one mechanism, and require the next evaluation to justify the added complexity.