Stage 0: What Is Relevance?

This chapter is the runbook’s front door: it explains how the runbook works, maps the pipeline the remaining chapters follow, and then does its own diagnostic job on the question every later chapter depends on.

About this runbook

This runbook is a diagnostic reference for existing retrieval and retrieval-augmented systems. Its conceptual framework is drawn from a practitioner course on search relevance, which the text refers to as the source course when attributing a claim specific to that material. It is organized by pipeline stage, one chapter per stage. The pipeline stages follow a six-part spine: a concept section, a symptom catalog, inspection guidance, diagnostic questions, a hypothesis-to-intervention map, and references. The three governed-search stages use a leaner variant suited to their incident-response character. Each section serves either a human diagnosing the system or an agent interrogating it.

The intended entry point is a symptom, because an engineer opening a runbook already knows something is wrong and usually does not yet know where. Start on the runbook overview by searching for the observed failure or filtering by its visible effect. Each result routes to the owning stage and exact catalog entry. Stable symptom IDs remain beside the reader-facing descriptions so incidents, links, and automated tools can refer to the same diagnosis without making the reader decode the identifier first.

A companion build guide lives at /build/. The two documents differ in kind: the build guide prescribes one concrete pipeline from scratch, while this runbook diagnoses whatever pipeline already exists. Stage chapters cross-link to the corresponding build chapters where a from-scratch reference implementation helps.

The pipeline map

The runbook models a retrieval-augmented system as ten operating stages plus this framing chapter. Offline stages prepare the corpus; online stages fire per query; evaluation and orchestration wrap around the core path; governed-search stages constrain evidence, policy context, and external boundaries.

#	Stage	What it covers
0	Framing: What is Relevance?	This chapter. The definition of relevance, the three contexts, how organizations measure search quality
1	Corpus & Indexing	Ingestion, chunking, analysis chains, inverted and vector index construction; the offline half of the system
2	Query Understanding	Classification, rewriting, expansion, entity recognition; everything that shapes the query before retrieval
3	Retrieval	Lexical, dense, and hybrid candidate generation; ANN index families; quantization
4	Ranking & Fusion	Score fusion, cross-encoder rerank, learning to rank, non-semantic signals (recency, popularity, locality)
5	Generation	Grounding an LLM answer in the candidate set; prompt construction; faithfulness
6	Evaluation & Feedback	Labeled sets, metrics, behavioral signals, click models, the reflected-intelligence loop
7	Agentic Orchestration	The agentic search loop; search as a tool; multi-step retrieval strategies
8	Permissioned Retrieval	Principal-aware document, row, and field eligibility before ranking; scoped caches and governed derivatives
9	Policy Through the Trajectory	Current authorization across branches, memory, caches, delegation, and checkpoint resume
10	Connectors and Trust Boundaries	Ingestion, identity, live-read, and action contracts; provenance, freshness, retries, and reconciliation

Multimodal retrieval, embedding finetuning, and observability cut across several stages. This runbook calls them out where they change a diagnosis rather than forcing them into one stage.

The stage map is also a routing table. Relevance failures usually present at stage 3 or 4, because that is where users see results, while the root cause is frequently elsewhere: a chunking decision at stage 1, a missing rewrite at stage 2, an absent feedback loop at stage 6. The symptom catalogs record where each failure presents and where its cause usually lives.

Concept

Relevance is a judgment made in context rather than a property stored with a document. The source course establishes this through counterexamples rather than a formula. The best hamburger restaurant in the world, in Tokyo say, is an irrelevant result for a hungry user standing on another continent. A hammer query on a music-store site fails however well the intent is understood, because no content exists to satisfy it. A basketball query on a news site is served badly by keyword match alone and well by keyword match combined with locality, recency, and popularity. In each case the document is unchanged; the judgment moves with the query, the person, and the moment.

The source course’s working model locates relevance at the intersection of three understandings. Content understanding is knowing what is actually in the index. Domain understanding is knowing what the entities, categories, and terms of art mean in this specific world. User understanding is knowing who is asking, where they are, and what they have done. Each context alone yields a degenerate system: content-only is plain keyword search, domain-only is a knowledge graph disconnected from data and users, user-only is collaborative filtering blind to content. The pairwise intersections carry the industry’s product names: content plus domain is semantic search, content plus user is personalized search, domain plus user is recommendations. The three-way intersection is user intent, which is the target the whole pipeline reaches toward.

This model motivates the source course’s strongest claim: no text-similarity algorithm can carry relevance alone. Domain-specific modeling far outperforms BM25 by itself, and the same limit applies to embedding similarity, because recency, locality, popularity, and conversion behavior are signals a text-matching score has no access to in principle. The illustrative form is the four-term scoring expression (keyword, geo-distance, recency, popularity) shown in Chapter 1, with the manual weights later replaced by learning to rank (stage 4).

From these pieces the relevance engineer’s job follows, and it is the job this runbook supports: state what relevance means for this system’s users, identify which of the three contexts the system models and which it lacks, choose techniques from the full toolbox rather than the fashionable subset, and close the loop so the system learns its definition of relevance from evidence. The scope point deserves emphasis because it is the source course’s thesis: AI-powered search covers rules-based, classical-ML, and deep-learning techniques, and the classical-ML zone (signals boosting, learning to rank, semantic knowledge graphs, query understanding) often delivers more retrieval value than deep-learning approaches applied directly.

Search systems often develop through the same sequence. A default retriever handles the head, long-tail failures accumulate as rules, the rules stop scaling, and the team invests in query understanding and learned ranking. Feedback is the step that makes those investments adaptive: capture the query, results, and user actions, then feed the evidence back into evaluation and ranking.

What good looks like at this stage: the organization can state, in terms a user would recognize, what a relevant result is; it can measure the system against that statement; and some feedback mechanism, however primitive, improves both over time.

Symptom catalog

response_latency_regression: Slow responses with no attributable stage

• Observable signal: Users report slow responses or tail latency regresses, while dashboards expose only one end-to-end timer. Engineers cannot separate queueing, retrieval, ranking, model time, retries, or parallel fan-out.
• Frequency: Common after adding generation, routing, or agent loops to a pipeline that previously measured only backend search latency.
• Cost of leaving unaddressed: Model upgrades, cache changes, and infrastructure work are chosen by guesswork. A small slow request class can remain hidden inside a healthy global median.

Begin with one request identifier and reconstruct the critical path. Record queue time, each stage and external call, fan-out and join boundaries, cache state, route and model, prompt and index versions, attempt count, terminal status, time to first token for streamed output, and total duration. Parallel child durations must not be summed as sequential work. The slowest required branch determines wall-clock latency.

Then compare distributions rather than isolated traces. Use the median (p50) for the common path and the 95th or 99th percentile (p95 or p99) for the tail, segmented by route, model, prompt version, index version, cache state, query class, attempt count, and status. The segment that moved identifies the owning stage. A slow retrieval call routes to Stages 1 through 4, generation time to Stage 5, and retries or routing overhead to Stage 7.

relevance_undefined: No operational definition of relevance

• Observable signal: Asked what a relevant result is for this system, different stakeholders give incompatible answers or none. Quality disputes settle by anecdote, seniority, or the most recent embarrassing query.
• Frequency: Very common. This is the default state of systems that grew from an out-of-box install.
• Cost of leaving unaddressed: Every downstream metric is uninterpretable. Teams cannot tell an improvement from a regression, so effort allocates by opinion.

Until this symptom is resolved, no other chapter of this runbook can be applied with confidence, because each chapter’s “what good looks like” presumes an answer.

no_labeled_queries: Nothing to measure against

• Observable signal: No file, table, or test fixture anywhere maps queries to known-relevant results. Retrieval and ranking changes ship on visual inspection of a few hand-picked queries.
• Frequency: Very common, including in systems with otherwise mature engineering practice.
• Cost of leaving unaddressed: Silent regressions. A change that helps ten visible queries and hurts a thousand invisible ones ships clean.

Even a small set helps. Fifty queries with judged results, maintained next to the code, converts relevance from opinion to measurement; stage 6 covers construction and metrics.

proxy_metric_optimization: Measuring something easier than relevance

• Observable signal: Reported search metrics are benchmark scores, engagement counts, latency percentiles, or corpus-quality figures. None of them would fail a result that is excellent in general and useless to this user now.
• Frequency: Common wherever no_labeled_queries holds, since proxies are what remain measurable.
• Cost of leaving unaddressed: Optimization effort flows to whatever the proxy rewards. Call this the Tokyo test: would the metric fail that Tokyo hamburger restaurant for a user standing in another country? A metric that would still rate it highly is measuring quality, and quality alone is a proxy.

missing_domain_signals: Relevance treated as an algorithm property

• Observable signal: The ranking function’s only input is a text-similarity score, BM25 or embedding cosine. Users see results that match the words and miss the situation: last year’s game for a query about tonight’s, a distant store for a local need, an abandoned product over a popular one.
• Frequency: Very common; this is the natural resting state after standing up a search engine or a vector database.
• Cost of leaving unaddressed: A hard ceiling. The source course’s claim is that domain factors (recency, locality, popularity, behavior) beat any algorithmic score on its own, so tuning the algorithm approaches the wrong limit.

The intervention path is incremental: add domain factors as weighted terms alongside the text score (the lesson’s example weights four terms equally), then let learning to rank set the weights once judgment data exists. Both live at stage 4.

embedding_conflation: The toolbox reduced to its newest shelf

• Observable signal: Roadmaps and design docs treat “AI search” and “embeddings plus LLMs” as synonyms. Every planned improvement is a model swap or a RAG variant; signals boosting, learning to rank, semantic knowledge graphs, and query understanding are absent as categories.
• Frequency: Common, and increasingly so; the popular discourse encourages the conflation.
• Cost of leaving unaddressed: The classical-ML zone is where the source course claims much of the affordable relevance value lives. An organization that cannot see the zone cannot compare against it, so it overpays for gains available more cheaply.

single_context_system: One context modeled, two missing

• Observable signal: The system’s behavior matches one of three degenerate profiles. Content-only: keyword or off-the-shelf vector search that treats every user identically and knows no domain entities. Domain-only: a knowledge graph or taxonomy effort disconnected from the index and from users. User-only: collaborative filtering with cold-start failures and blindness to what items actually contain.
• Frequency: Common; each profile is a reasonable starting point that was never extended.
• Cost of leaving unaddressed: The failure modes of the missing contexts, permanently. A content-only system cannot serve “software engineer” differently to a Chicago user; a user-only system cannot recommend anything new.

Diagnosis is a routing exercise: name the missing contexts, then invest by stage. Content understanding is stages 1 and 3; domain understanding is stage 2 and the knowledge-graph material; user understanding is stage 6’s signals and personalization.

patchwork_rules_sprawl: Relevance by accumulated exception

• Observable signal: Large synonym files, pinned-result lists, and per-complaint boost rules, accumulated over years, with no owner, no tests, and no record of why individual entries exist. New complaints are fixed by adding entries.
• Frequency: Very common in systems more than a couple of years old; this is the second stage of the evolution arc.
• Cost of leaving unaddressed: The rule pile grows monotonically because nobody dares remove an entry, rules conflict silently, and prioritization defaults to whoever complains loudest, which makes the loudest stakeholder the de facto definition of relevance.

The arc’s resolution is the move from patching outputs to understanding queries: classification, concept expansion, automatic rewriting. That is stage 2’s subject. The patchwork itself has diagnostic value on the way out, since it is a hand-built record of the system’s long tail.

no_feedback_loop: The system never learns

• Observable signal: Query, result, and user-action data are either not logged or logged and consumed by nothing. Queries that failed a year ago fail identically today. Nobody can say when quality last improved without a human shipping a change.
• Frequency: Very common outside large e-commerce and web-search organizations.
• Cost of leaving unaddressed: A structural cap on quality regardless of algorithmic sophistication. The system repeats the same ranking decisions after user behavior shows that they are failing.

The loop’s shape is simple to state: log query, results, and actions; process the signals; learn a relevance model; feed it back into serving. The techniques that fill in the shape (signals boosting, click models, learning to rank, active learning, A/B testing, bandits) are stage 6’s catalog. Logging should start immediately even if nothing consumes it yet; the history becomes training data later.

How to inspect

Auditing an organization’s relevance definition means reading its code for what actually reaches the ranking function and reading its practices for how quality claims get settled.

By code pattern

Pattern or location	What you’re looking for	What suggests a problem
Ranking / scoring function inputs	Every signal that reaches the final score	Only a text-similarity score (BM25 or cosine) reaches it: missing_domain_signals
synonyms.txt, elevate/pin configs, boost-rule files	Size, age, ownership of the rule pile	Hundreds of entries, no owner, no tests, git history of per-complaint additions: patchwork_rules_sprawl
Test fixtures, eval/ directories, golden-query files	Any query-to-relevant-result mapping	None exists, or it exists and no CI job or release gate reads it: no_labeled_queries
Analytics and logging calls in the search path	Query, result, and click/action logging	No logging, or logs written and never consumed: no_feedback_loop
Retrieval dependencies and imports	The shape of the toolbox in use	Vector DB and embedding client with no lexical index, and no LTR or signals code anywhere: embedding_conflation, often with vector_dominance (see retrieval#vector_dominance)
User attributes in the query path	Whether location, history, or segment reach retrieval or ranking	No user attribute is ever consulted: user context is unmodeled (single_context_system)
Entity, taxonomy, or domain-vocabulary references	Whether domain knowledge connects to the index	A knowledge graph or taxonomy exists in a silo no query ever touches: domain context unmodeled (single_context_system)
Request tracing and timing hooks	One request id, stage spans, fan-out and join boundaries, route and version dimensions	One total timer, missing model or cache dimensions, or no time-to-first-token measure: response_latency_regression

By measurement

Framing-level measurement is mostly about whether measurement exists at all. Healthy-range thresholds below are judgment calls; later chapters supply stage-specific numbers.

Measurement	How to measure	Healthy range / red-flag threshold
Labeled query set: existence, size, freshness	Locate it; count queries; check last-modified against last ranking change	Red flag: absent, or older than the last significant ranking change
Definition consistency	Interview 3-5 stakeholders: “what is a relevant result here?”; compare answers	Red flag: incompatible answers, or answers that describe document quality with no user context
Signal inventory	Count distinct non-text signals reaching the final score (recency, locality, popularity, behavior)	Zero is a red flag for any system past its first year
Change-validation practice	For the last 5 relevance-affecting changes, find what evidence gated each	Red flag: visual inspection of hand-picked queries only
Repeated-failure rate	From query logs, find queries that fail (abandonment, reformulation) the same way across months	Persistent identical failures indicate no_feedback_loop
Rule-pile trajectory	Count synonym/boost/pin entries over time from version control	Monotonic growth with no deletions suggests patchwork_rules_sprawl
Latency by critical-path segment	Compute p50/p95/p99 for queue, retrieval, ranking, generation, retries, and total duration; segment by route and version	Any segment moving with the regression identifies an owner; a flat global median with a rising route-specific p99 still fails

Diagnostic questions

Triage flow for an agent or engineer opening an unfamiliar system. Earlier questions are coarse; branches go deeper. This flow is also the recommended entry into the runbook as a whole.

1. [interview] What does a relevant result mean for this system, in terms a user would recognize? Force a concrete answer (“a local, recent article for a news query”; “the product the user buys, in the top 3”). No answer, or answers that vary by stakeholder: relevance_undefined; resolve before proceeding.

2. [code] Does a labeled query set exist, and when did it last gate a change? Absent: no_labeled_queries; route to stage 6 for construction. Present but stale or unenforced: treat as absent.

3. [interview] What metrics are reported for search, and would they fail a result that is excellent in general but useless to this user now? Metrics that would not fail it are measuring quality rather than relevance: proxy_metric_optimization.

4. [code] Can one slow request be reconstructed as a critical path? If only total duration exists, add stage spans before changing models or infrastructure. If stage spans exist, segment p50 and tail latency by route, model, prompt, index, cache state, attempt count, and status, then route the diagnosis to the stage whose time moved.

5. [interview] Which of the three contexts does the system model: content, domain, user? For each claimed context, ask for the code path where it reaches retrieval or ranking. One context only: single_context_system; route by the missing contexts.

6. [code] What signals reach the final ranking score? Text similarity only: missing_domain_signals; route to stage 4. Multiple signals with hand-set weights: ask when the weights were last validated, and whether LTR has been considered.

7. [code] What does the rule pile look like? Count synonym, boost, and pin entries; check ownership and tests. Large, unowned, growing: patchwork_rules_sprawl; route to stage 2.

8. [interview] When a user fails to find something and then finds it another way, does the system ever learn from that? No logging, or logs nothing consumes: no_feedback_loop; route to stage 6. If logging exists, ask what model or process consumes it and how often.

9. [interview] What options were considered for the last major search investment? All candidates were embedding or LLM variants: embedding_conflation; introduce the classical-ML zone before the next investment decision.

Interventions by likely cause

Ordered by frequency-of-cause. This stage’s interventions are mostly routing: the frame identifies the gap, and a later stage owns the fix.

Hypothesis	Supporting evidence	Intervention	Expected impact	Cost / risk
Relevance was never operationally defined (framing#relevance_undefined)	Stakeholder answers incompatible or absent	Write a one-page relevance definition in user-recognizable terms; get stakeholder sign-off; derive metrics from it	Every later diagnosis becomes interpretable	Low: a workshop and a document
Nothing is measured against known-good results (framing#no_labeled_queries)	No labeled set found; changes gated by inspection	Build a small labeled query set (50-100 queries with judged results); wire it into release checks; grow it from real query logs	Converts relevance work from opinion to measurement	Low to medium: labeling effort; see stage 6 for method
Reported metrics are proxies (framing#proxy_metric_optimization)	Metrics fail the Tokyo test	Re-anchor reporting on labeled-set relevance metrics; keep proxies as secondary diagnostics	Optimization effort realigns with user-perceived quality	Low: measurement work, plus organizational buy-in
Latency has no attributable stage (framing#response_latency_regression)	One total timer; no route, version, retry, cache, or time-to-first-token dimensions	Add request-scoped stage spans and fan-out/join boundaries; compare p50/p95/p99 by route and version; route the fix to the stage whose time moved	Slow requests become attributable before an expensive intervention is chosen	Low to medium: trace plumbing and retention controls
Ranking sees only text similarity (framing#missing_domain_signals)	Score function inputs are BM25/cosine only	Add domain factors (recency, locality, popularity) as weighted score terms; plan LTR to learn the weights	The source course’s claim: dominates any algorithm-only tuning	Medium: signal plumbing; see stage 4
No feedback loop exists (framing#no_feedback_loop)	No query/result/action logs, or unconsumed logs	Start logging query, results, and actions now; adopt signals boosting as the first consumer; grow toward LTR and click models	Removes the structural cap on improvement; compounding	Medium: logging is cheap, the consuming pipeline is ongoing work; see stage 6
Rule patchwork substitutes for query understanding (framing#patchwork_rules_sprawl)	Large unowned rule pile, growth per complaint	Freeze the pile; mine it as a record of the long tail; invest in query classification and rewriting	Sustainable replacement for per-complaint fixes	Medium to high: query-understanding infrastructure; see stage 2
A whole context is unmodeled (framing#single_context_system)	System matches a degenerate profile	Name the missing contexts; route: content to stages 1 and 3, domain to stage 2, user to stage 6	Removes an entire class of failures per context added	High: each context is its own data and modeling investment
The toolbox is reduced to embeddings and LLMs (framing#embedding_conflation)	Roadmap is all model swaps and RAG variants	Evaluate classical-ML options (signals boosting, LTR, SKG, query rules) against the next planned deep-learning investment	Frequently more relevance per unit cost	Low to evaluate; interventions carry their own costs