Chapter 7 made search quality and uncertainty observable, giving the system evidence it can use while the user is still waiting. When a search for a family movie about food returns films with food somewhere in their plots but no result the system can verify as a family choice, the system can act on that evidence by deciding whether to answer from incomplete results, search again, or stop.
The opening instrument replays a recorded attempt to answer that request, using result lists produced by searches against the 250-film index. The thoughts and reflections were written to explain what happens at each step rather than expose hidden model reasoning. Run the sequence or advance it one phase at a time, watching how the first results leave part of the request unresolved, the revised query changes the search strategy, and the record lookup tests what the system can support.
That ability to inspect an intermediate result and choose another bounded action is agentic orchestration. A one-pass retrieval-augmented generation pipeline retrieves context and answers from one assembled evidence set. Orchestration adds a control layer around that pipeline: when a sufficiency check finds that the retrieved evidence does not cover the request, the system can spend a bounded retry on another query or tool call instead of answering immediately. A trajectory is the complete record of one run, including the queries issued, tools called, evidence returned, decisions made, and condition that ended the loop.
When the first result is incomplete
The first search retrieves films centered on food, but the returned titles, summaries, and scores cannot establish whether any candidate suits a family audience. A sufficiency check like the one in Chapter 6 can expose that uncovered constraint by comparing the results with the whole request, while the surrounding application code catches narrower failures such as an empty result set, scores below a floor, or a required field missing from every candidate.
Together, those checks turn a vague sense of poor quality into a specific evidence gap. The next action should try to close that gap, and a bounded loop should stop or ask for clarification when its connected tools cannot do so. Otherwise, another search merely gives the model more material from which to write an unsupported answer.
Search, inspect, and retry
The first results leave one unresolved job: verify that a candidate suits a family audience. The model responds by narrowing the search to chefs, kitchens, and restaurants, but that query changes the kind of food film being retrieved without asking for the missing audience evidence. The harness, the application code that manages tools and state, validates the query and runs vector search; when Ratatouille surfaces, the model has a specific record to inspect rather than a title from which to guess.
The lookup confirms a plot centered on a chef, cooking, and a restaurant kitchen, but its genres are comedy and drama, and its summary doesn’t include an age rating or other evidence about suitability for children. The record cannot support the instrument’s final claim that children can watch the film. A reliable loop would disclose that gap, ask what the user means by a family choice, or request permission to consult a source that records audience suitability.
The revised search produced a different food match without resolving the failed constraint, which is the kind of motion an agent can mistake for progress. Preventing that mistake requires a quality signal that compares each attempt with the full request and application rules that limit which actions may run, how much they may spend, and when they must stop. Those controls place evaluation at two boundaries around the model.
Put evaluation around the model
Inside the loop, a judge compares each attempt with the full request. Depending on the evidence available, that comparison may combine required-field checks with candidate scores from the cross-encoder or learning-to-rank model in Chapter 5, or use a separate language-model call. Any score for the chef-and-kitchen results supports that direction only after the judge has been calibrated to the request’s constraints, and no judge can manufacture a missing family signal from a plot record that lacks it.
When the model stops requesting tools and proposes an answer, the harness evaluates the complete record of queries, tool calls, evidence, and decisions that produced it. The harness may accept the answer, return feedback for a bounded retry, or stop when the process reaches a retry ceiling, exhausts its token or latency budget, or stops improving according to the judge. The judge guides a search still in progress, while the harness retains authority over execution and completion; the judge’s scores and the harness’s acceptance rules both need validation against human judgments and outcomes using the discipline from Chapter 7.
Follow the figure below from the model’s proposal through harness validation, tool execution, and judging. The boundary shows why the model can neither run its own action nor certify its own answer.
The figure also exposes the limit in the movie trajectory: the loop can keep searching, but only a permitted tool that returns audience evidence can resolve suitability. Tool design sets both the actions the model can take and the evidence gaps it can close.
Give the loop tools that already know the domain
For the movie task, search accepts keywords and returns titles, summaries, and scores, while lookup accepts a film identifier and returns its record. These fixed inputs and outputs support discovery and plot verification without requiring the model to construct backend queries. Because neither tool returns audience evidence, calling them again cannot establish the remaining constraint.
Every backend decision the tool leaves to the model becomes another decision the model must make while searching. Exposing an engine-specific query language forces it to manage syntax and domain rules, while high-level parameters and fixed lists of valid categories remove those failure points before retrieval begins. Once the call is valid, the returned results determine how much work remains: weak retrieval forces the model to reformulate and try again, spending more tokens and time on each attempt.
Because weak retrieval makes every retry slower and more expensive, the agent should begin with the production search engine when it already searches the needed content and applies the required filters and business rules. If that engine’s ranking has been tuned against relevance judgments and user feedback, wrapping it as a tool preserves that work. Replacing it with newly chunked documents, embeddings, and a vector database creates a second retrieval system whose relevance, monitoring, and operating behavior must be established again.
Wrapping the production engine as a tool changes who chooses the query without discarding how the engine retrieves and ranks the results. Each reformulated query still passes through its keyword or vector retrieval, taxonomy rules, rerankers, and learning-to-rank models, preserving the ranking behavior from Chapter 5 and the feedback incorporated through Chapter 7. The missing family signal marks the boundary of that reuse: when the production stack does not contain the evidence a request requires, the agent needs a new source or index rather than another reformulation.
Use trajectories to improve the next request
A retry can repair one request, while its trajectory may expose a problem shared by many requests. The movie trajectory records each query, tool call, judge score, returned result, and unresolved constraint so the agent can decide whether to search again; aggregated across sessions, those same records let the feedback system from Chapter 7 distinguish recurring failures in query strategy, tool design, retrieval, ranking, or evidence coverage.
Repeated broad food results point toward the search tool, retrieval rules, or ranking features, while repeated endings without audience evidence reveal a different failure: the system needs a governed audience field or another source whose contract and freshness support that claim. Fixing the diagnosed component gives the next session a better starting point and should reduce the retries needed to reach a supported answer.
The request-time loop and the across-session feedback loop connect recovery to the feedback path introduced in Chapter 1: bounded retries work on the current request, while measured outcomes guide changes to the components used by future requests. Trajectories remain evidence rather than automatic training labels, so teams must validate their interpretations against human judgments and outcomes, correct for bias, version each change, and test it offline and online before release.
The loop developed here handles one request, one sequence of tool calls, and one decision to continue or stop. When a request contains several information needs that depend on one another, the system cannot choose the next action until it identifies the required tasks and their dependencies. Chapter 9 extends the loop by decomposing that request, routing each part to a source that can answer it, and deciding which searches should run in sequence or in parallel.