Ask a movie search engine for “shark attacks a beach town” and the relevant result seems clear. Because the plot of Jaws directly matches the query, a basic ranker can place it first and an evaluator can support that choice from the film summary. Change the request to “best movie for a rainy date night” and the same evidence no longer settles the question. The corpus can offer plausible films, but a plot summary does not describe this viewer’s mood, personal context, or current streaming options.
That gap can hide inside a good evaluation score. Suppose a team judges search results using the same plot summaries. Its evaluation set will reward strong plot matches because those are the choices its evaluators can defend from the evidence they have. A new ranker may agree with those recorded judgments more often, yet still struggle with requests that depend on evidence the set never captured.
After release, explicit plot queries can continue to work while situational queries return films that appear reasonable but fail the user’s task. Query reformulation or abandonment may expose the omission if the team examines those outcomes. Until then, the offline evaluation tells a reassuring but incomplete story: the model has become better at reproducing a definition of relevance built mostly from topical similarity.
Before choosing a retrieval architecture, ranking model, or evaluation framework, the team therefore needs to decide what a useful result means for the people using the system. For each request, it must ask which result would solve the user’s problem now. That decision is the relevance judgment the rest of the system will try to reproduce.
Relevance is a judgment
The judgment cannot be derived from a plot summary alone. Jaws fits the request for a shark attacking a beach town, but nothing stored in that summary establishes whether the film suits this viewer’s evening or is available where they live. Even the words in the query can leave room for several reasonable answers. Two people may search for “alien movie” while wanting different experiences, horror for one and first contact for the other.
Relevance is this assessment of how well a result fits a person’s goal and circumstances at the time of the request. The system’s corpus, the collection of items it can search, provides evidence about each film. It cannot declare one film relevant for every future request because the request supplies part of the meaning. Someone must interpret the available evidence in light of that request and its context.
The instrument below asks you to perform that work using the 250-film corpus explored throughout this book. Each round presents evidence from its original plot summaries and normalized metadata, then asks how far that evidence answers the request. Choose the result you judge more relevant, then compare the evidence behind your choice.
Watch what changes across the three rounds. “Shark attacks a beach town” gives you a clear criterion, and the summaries support one strong answer. “Best movie for a rainy date night” introduces a preference the corpus does not record. “Alien movie” leaves the desired genre and experience unstated, so several choices can be defended from the same evidence.
That disagreement is useful. It reveals where the request or the available evidence permits more than one reasonable judgment. An offline measure such as normalized discounted cumulative gain (nDCG) reports how closely a ranked list agrees with a particular set of judgments. Its score carries the assumptions and preferences of the people who supplied those judgments, so an improvement means the system matches that recorded view more closely. Chapter 7 explains how to collect judgments at scale and decide whose perspective they should represent.
A judgment can use only the evidence that reaches it
The date-night judgment can improve only when the missing evidence reaches the system. Three sources recur throughout this course, and each contributes a different part of the answer: content, domain, and user context.
Content is what the system can search, including each item’s text and fields. For the date-night request, it can establish that a film is a comedy, a horror story, or a romance. Domain adds the vocabulary, categories, relationships, and business rules needed to interpret that content. It can connect “date night” with useful genre or tone labels even when those exact words never appear in a plot summary. User context describes the person making the request and the situation around it, including location, permissions, history, and activity in the current session. It can narrow the choice to films this person can watch and is likely to enjoy now.
Together, these sources allow the system to estimate user intent, which in this course means its best supported account of the task behind the request. A weakness in any source limits the judgment the ranker can make. Rich content cannot establish local availability without location and catalog data, while detailed user history cannot recover a film whose content was never represented well enough to retrieve.
The same framework can diagnose the opening failure. List the content, domain, and user signals that reach the ranking function, then compare them with the conditions that would make a result useful. Keyword and vector retrieval can widen the view of content, domain features can add vocabulary and constraints, and personalization can add evidence about the user. Each technique reveals a different part of the decision. None can rank by availability, recency, permissions, or the current task until those signals enter the path.
Relevance depends on more than text similarity
Text similarity is one way to use the content source, and the opening example shows what happens when that signal stands in for the whole judgment. Similarity remains useful because it identifies results that discuss the subject of the query. Once several results clear that bar, other evidence may be needed to decide which one best fits the user’s task and circumstances.
Two approaches taught later in this course illustrate the boundary. BM25, a keyword ranking function, scores terms shared by the query and document. Vector search compares numerical representations of their meaning. Both can estimate how closely a result relates to the query. Neither can reliably determine from the text alone whether an item is available, a document is current, or a result is appropriate for this user.
Consider a news site responding to the query “basketball.” Text matching may find thousands of articles about the sport, which solves the first problem but leaves the system with too many plausible results. A recent story may be more useful than an old one, a nearby team’s game may matter more to this reader, and an article receiving sustained attention may deserve a boost. The ranking function can bring those signals into the same decision:
The equal weights are placeholders rather than a claim that every signal matters equally. A production system must learn or set how much each one should contribute, then revisit that relationship as users, content, and business conditions change. Chapter 5 shows how evidence can be used to learn these weights instead of fixing them once by hand.
Before any weight can help, recency, proximity, and popularity must be collected, represented, and supplied to the ranking function. Improving relevance often begins with that practical inventory: what would help the decision, and which part of that evidence can the system see today?
From isolated fixes to a learning system
Most search systems begin with the evidence that is easiest to represent. A baseline may rank with keywords or vectors and perform well on clear, common queries because the content contains enough information to support the judgment. Its limits emerge in less common requests, ambiguous language, and situational constraints, where the query and the indexed content no longer tell the whole story.
Teams usually meet those failures with local fixes. They add a synonym, pin a result, or write a business rule for a visible complaint. Each fix preserves something the team has learned about its domain or users, but that knowledge remains scattered across exceptions. As the rules accumulate, they become harder to reconcile, and the loudest complaint can receive more attention than a quieter failure affecting many more requests.
Repeated exceptions eventually point to reusable capabilities. Instead of patching one query, the system can classify intent, expand a domain concept, rewrite a request, or personalize a result so that the missing context becomes available to a whole class of queries. Chapter 3 develops these techniques, which turn recurring observations into evidence the system can apply again.
The next change is to let outcomes influence those capabilities. Relevance judgments, behavioral signals, controlled experiments, and learned ranking can show whether a change actually helped people complete their tasks. Once that loop exists, the system can adjust from observed results, provided the feedback represents the outcome the team cares about rather than a convenient substitute.
Some systems go further and use bounded automation to identify failures, propose improvements, and evaluate the results with less manual intervention. Human judgment still sets the objective and the limits because feedback remains incomplete, user needs change, and business constraints determine which tradeoffs are acceptable.
This development is a recurring response to pressure rather than a required maturity sequence. A growing collection of handwritten rules suggests that isolated fixes should become systematic query or ranking features. A sophisticated retrieval pipeline whose observed outcomes never change a ranking decision has reached a different limit: it needs a feedback loop. In either case, the next improvement depends on turning a lesson about relevance into evidence the system can use.
Prepare evidence offline, apply it online
Once the team knows which evidence it needs, it must decide when that evidence can be prepared. Production search systems separate work completed before a request from work that must happen while a user waits. This course calls them the offline and online paths. The map below opens on the online path; switch to the offline tab to see what must already be ready before the first request arrives.
Process one request under a latency limit
Use the prepared artifacts to turn the query and current context into a small, ordered result set while the requester waits.
- 1
Interpret
Normalize the query and apply eligibility filters.
- 2
Retrieve
Find a broad set of plausible candidates.
- 3
Decide
Enrich, score, and order the small set that remains.
Offline work prepares evidence the system can reuse. It defines document structure and text transformations, builds indexes and numerical representations, creates ranking features, trains models, and assembles evaluation judgments. These artifacts must be versioned and compatible because the online path depends on them agreeing about how documents, queries, and features are represented.
Online work applies those artifacts to one request. It combines the query with current user and session context, filters out ineligible results, retrieves candidates, adds any signals available at request time, and scores the remaining results. Each step spends part of a latency budget, so cheaper operations handle broad candidate sets before more expensive models examine a smaller group.
Agentic search operates around this online path. An agent can inspect an initial result, reformulate the query, retrieve again, or decide that it has enough evidence to answer. Those choices can make better use of the available system, but they still depend on the corpus, indexes, filters, and rankers prepared offline. If the underlying system never collected or exposed a piece of context, another search step cannot reliably recover it.
The two paths meet again through feedback. Actions observed online become evidence for offline evaluation and training, which can change the artifacts used by later requests.
Feedback changes the system
A feedback loop begins when activity from the online path returns as evidence for offline evaluation. The system records the query, the context available at the time, the results shown, and what happened next. With that history, the team can compare its original relevance judgments with the outcomes those judgments produced, then revise the evaluation set, create ranking features, adjust weights, or train a new model. AI-Powered Search calls this process reflected intelligence.
The date-night request shows what the loop can reveal. A ranker may improve nDCG by reproducing plot-based judgments more closely, while people searching for situational recommendations rewrite their queries more often or leave without choosing a film. Those outcomes do not prove that the new ranker is worse, but they show that the higher offline score is not enough to establish that search became more useful. The ranker may have produced better plot matches without becoming better at helping people choose something they could and wanted to watch.
Understanding that difference requires the team to follow individual sessions. A viewer who opens a film, spends several minutes on its page, and then rewrites the query has left a trail of evidence, but the trail does not explain itself. The viewer may have liked the film before discovering that it was unavailable, clicked because the title or artwork stood out, or refined the query as a normal part of choosing. The ranker also shaped the session by deciding which films appeared and where they appeared.
Before this behavior can change the system, the team must decide which interpretations the recorded events can support and which remain uncertain. Chapter 7 develops that work through explicit judgments, controlled experiments, and safeguards against behavioral bias. Once the evidence has a defensible meaning, the signal boosting and learned ranking methods in Chapter 5 can use it. Ignoring the evidence leaves the existing definition of relevance unchanged, while interpreting it poorly can reinforce the same mistake at greater scale.
Follow the evidence through the system
The rest of the course follows this evidence as a request becomes a ranked result and, eventually, an answer. Part I builds the search pipeline one stage at a time. Chapter 2 determines which content can enter the system and how it is represented. Chapter 3 interprets the incomplete language of a request, Chapter 4 finds plausible candidates, and Chapter 5 orders them using the signals chosen to represent relevance. Chapter 6 turns the surviving evidence into an answer, while Chapter 7 tests the result against explicit judgments and observed outcomes.
Part II lets the system inspect and adapt that path. Beginning with Chapter 8, the system can decide whether one search produced enough evidence, break larger requests into tasks, preserve state across attempts, recover from failures, authorize proposed operations, and evaluate the complete trajectory. Part III carries authority through the same path by controlling which evidence may enter retrieval, which actions remain allowed as conditions change, and how external connectors affect the result.
Two companion sections support different kinds of work. The runbook traces production symptoms to the pipeline stages and missing signals that can cause them. The implementation guide builds a runnable version of the system in plain Python.
As each new technique appears, ask two questions: what evidence can it see, and what definition of a useful result is it optimizing? Those questions keep every later implementation choice connected to the relevance judgment established here, beginning with the content the system chooses to preserve.