A search team updates the system that orders results. In offline tests, the new ranking model improves normalized discounted cumulative gain (nDCG), a metric that rewards placing relevant results near the top. After release, users find that the top results match their search terms but are outdated, unavailable, or otherwise unsuitable for what they are trying to do. They return to the search box and revise their queries to compensate, often adding constraints the system failed to infer. These repeated attempts lead to more support tickets, while conversion falls on the queries the change was intended to improve.
The ranking model did what the evaluation rewarded: it moved topically similar results higher. But the judgments behind the metric did not distinguish between a result that matched the query and one the user could act on. They omitted whether the result was current, available, and suited to the user’s task. The offline score improved while the search experience worsened.
That gap is the starting point for this chapter. Before choosing a retrieval model, ranking function, or evaluation metric, a team needs a definition of relevance that reflects what users are trying to accomplish. For any request, the first question is: what would make a result useful to this user, in this situation, now?
Relevance is a judgment
That question cannot be answered from the document alone. A result may match the query exactly and come from an authoritative source, yet still be unusable because the requester cannot open it, the information is stale, the item is unavailable, or the result applies to a different tenant or software version.
The user’s task creates another source of variation. Someone searching for an error code may want an explanation, the incident that fixed it, or the service currently emitting it. Each task makes a different result useful, even though the query text stays the same.
Relevance is therefore a judgment made at query time about the fit between a result and a person’s goal and circumstances. The corpus can store evidence about a result, but it cannot declare that result relevant for every future request. Evaluating a search system requires someone to turn the request and its context into judgments. Try making a few yourself, and notice how far the query alone gets you.
Watch what changes across the three rounds. “Shark attacks a beach town” gives you a clear criterion, and the plot summaries support one strong answer. “Best movie for a rainy date night” introduces a preference the corpus does not record. “Alien movie” leaves the desired genre and audience unstated, so several answers are defensible.
Those differences become part of the measurement. An offline score reports how well the system agrees with a particular set of judgments, including the assumptions and preferences of the people who supplied them. Chapter 7 covers how to collect those judgments at scale and decide whose perspective they should represent.
A judgment can use only the evidence that reaches it
Those judgments can draw only on the evidence the system supplies. Three sources recur throughout this course: content, domain, and user context.
Content describes what can be searched: the documents or items, their text, and their fields. It provides the evidence that a result matches the request. Domain supplies the meaning and rules specific to the setting, such as its vocabulary, categories, relationships, and business constraints, and helps the system interpret that evidence correctly. User context describes the person making this request and the situation around it, including permissions, location, history, and activity in the current session. It determines whether a result is accessible, appropriate, and useful for this person now. In this course, user intent refers to the system’s best estimate of that person’s task based on the evidence it has. Weakness in any one source limits the judgment the ranking model can make.
This framework also helps diagnose failures. List the content, domain, and user signals that actually reach the ranking function, then compare them with the conditions that make a result useful. Different techniques can only widen different parts of that view: keyword and vector retrieval read content, domain features add vocabulary and constraints, and personalization adds the user. If signals about availability, recency, permissions, or the current task never reach the model, the model cannot rank by them. That was the failure in the opening example.
Relevance depends on more than text similarity
Text similarity is one content signal, and the opening example shows what happens when it stands in for the whole judgment. Similarity is still useful. It identifies results that discuss the subject of the query, but it cannot determine on its own which of those results best fits the user’s task and circumstances.
Two approaches taught later in this course illustrate the boundary. BM25, a keyword ranking function, scores terms shared by the query and document. Vector search compares numerical representations of their meaning. Both can estimate how closely a result relates to the query. Neither can reliably determine from the text alone whether an item is available, a document is current, or a result is appropriate for this user.
Consider a news site and the query “basketball.” Text matching may find thousands of articles about the sport. Ordering those articles requires additional judgments. A recent story may be more useful than an old one, a story about a nearby team may matter more to this user, and an article receiving sustained attention may deserve a boost. The ranking function can combine those signals explicitly:
The equal weights are placeholders. A production system must decide how much each signal should contribute, and those relationships change as users, content, and business conditions change. Chapter 5 shows how to learn the weights from evidence rather than setting them once by hand.
Recency, proximity, and popularity must be collected, represented, and supplied to the ranking function. Improving relevance often begins by identifying which useful signals the system cannot currently see.
How search systems develop
Search systems tend to develop by using more evidence and becoming more systematic about learning which evidence matters. We can group that development into five stages.
The first stage is a baseline search engine. It may use keywords or vectors, but it primarily ranks from content and often performs well on clear, common queries. Its limits appear in the long tail: the less common requests, ambiguous language, and situational constraints the baseline does not handle.
In the second stage, the team fixes those failures one at a time, adding synonyms, pinning results, and writing business rules. Each fix encodes something useful about the domain or the user, but the knowledge stays scattered across exceptions that grow harder to reconcile over time, and the most visible complaint can draw more attention than the most common failure.
The third stage turns repeated exceptions into reusable capabilities. The system classifies query intent, expands domain concepts, rewrites queries, and personalizes results. Instead of patching one query, the team tries to represent the missing context in a form that can improve a class of queries. Chapter 3 develops these techniques.
In the fourth stage, the team builds feedback loops. Relevance judgments, behavioral signals, A/B tests, and learned ranking provide evidence about whether changes help users complete their tasks. The system can now adjust from observed outcomes, provided that the feedback represents the outcome the team actually cares about.
The fifth stage is a system that continuously identifies failures, chooses improvements, and evaluates the results with limited manual intervention. Few systems reach that point. Feedback is incomplete and biased, user needs change, and business constraints still require human decisions about what the system should optimize.
These stages describe recurring constraints rather than a mandatory sequence. If you run a search system today, you can probably place it: count your hand-written rules and ask whether any outcome you observe ever changes a ranking decision. A growing collection of manual rules suggests that isolated fixes should become systematic query or ranking features. A sophisticated retrieval pipeline that cannot learn from outcomes needs a feedback loop.
Prepare evidence offline, apply it online
The evidence used to rank a result is prepared and applied at different times. Production search systems separate work done before a request from work that must happen while a user waits. The course refers to these as the offline and online paths. The map below shows the two paths. It opens on the online path; switch to the offline tab to see the work that must already be finished before the first request arrives.
Process one request under a latency limit
Use the prepared artifacts to turn the query and current context into a small, ordered result set while the requester waits.
1 · Interpret
Normalize the query and apply eligibility filters.
2 · Retrieve
Find a broad set of plausible candidates.
3 · Decide
Enrich, score, and order the small set that remains.
Offline work prepares the evidence the system can reuse. It defines the document structure and text transformations, builds indexes and numerical representations, creates ranking features, trains models, and assembles the judgments used for evaluation. These artifacts must be versioned and compatible because the online path depends on them agreeing about how documents, queries, and features are represented.
Online work applies those artifacts to one request. It combines the query with current user and session context, filters out ineligible results, retrieves candidates, adds any signals available at request time, and scores the remaining results. Each step spends part of a latency budget, so cheaper operations handle broad candidate sets before more expensive models examine a smaller group.
Agentic search operates around this online path. An agent can inspect an initial result, reformulate the query, retrieve again, or decide that it has enough evidence to answer. Those decisions can make better use of the available system, but they still depend on the corpus, indexes, filters, and rankers prepared offline. An agent cannot reliably recover context the underlying system never collected or exposed.
The two paths meet again through feedback. Actions observed online become evidence for offline evaluation and training, which can change the artifacts used by later requests.
Feedback changes the system
A feedback loop turns outcomes from one request into evidence for improving later requests. The system records the query, the context available at the time, the results shown, and what happened next. The team can then use those records to revise evaluation judgments, create ranking features, adjust weights, or train a new model, a process AI-Powered Search calls reflected intelligence.
The opening example shows why the loop matters. Offline nDCG improved, but users reformulated more often, contacted support, and converted less often. Those online outcomes revealed that the evaluation set rewarded topical matches while overlooking whether users could complete their tasks.
Behavioral signals still require interpretation. A click may indicate interest, confusion, or an attractive title. Long dwell time may mean that a result was useful or difficult to understand. A reformulation may indicate failure, or it may be a normal step in exploration. Position, presentation, and existing ranking choices also influence what users can click. Feedback becomes useful only after the team decides what each signal can support and accounts for these sources of bias.
Chapter 7 develops evaluation and feedback collection, including controlled experiments and the limits of behavioral evidence. Chapter 5 shows how signals boosting and learned ranking use that evidence. For now, carry one connection forward: online behavior can challenge the judgments prepared offline. If the team ignores that evidence, the system keeps optimizing its existing definition of relevance. If the team interprets it poorly, the feedback loop can reinforce the same mistake at greater scale.
How the chapters are organized
The chapters follow how a request becomes a relevance judgment and, eventually, an answer. Part I builds the search pipeline one stage at a time. Each chapter examines what evidence a stage creates, what evidence it can use, and how its decisions constrain the stages that follow.
Chapter 2, Corpus and Indexing, determines what content the system can retrieve and how that content is represented. Chapter 3, Query Understanding, interprets incomplete query text using domain and user context available before retrieval. Chapter 4, Retrieval, uses lexical and vector methods to produce candidate results, with different strengths and failure modes. Chapter 5, Ranking and Fusion, combines candidates and orders them using the signals that represent the team’s definition of relevance.
Chapter 6, Generation, turns retrieved evidence into an answer while preserving its connection to the source material. Chapter 7, Evaluation and Feedback, tests retrieval and generation against explicit judgments and observed outcomes.
Part II begins with Chapter 8, Agentic Orchestration, which lets the system inspect results, decide whether the available evidence is sufficient, and search again when one attempt is not enough. Chapter 9 expands that loop to decomposition and tool routing. Chapter 10 follows the state an agent needs across attempts and the recovery paths it needs when tools fail. Chapter 11 authorizes proposed operations. Chapter 12 evaluates complete trajectories, including their outcomes, latency, and cost. Part III then carries authorization into retrieval, trajectory state, and connector boundaries.
Two companion sections support different kinds of work. The runbook traces production symptoms to the pipeline stages and missing signals that can cause them. The implementation guide builds a runnable version of the system in plain Python.
As each new technique appears, ask two questions: what evidence can it see, and what definition of a useful result is it optimizing? Those questions connect every later implementation choice to the relevance judgment established in this chapter.