Chapter 11 ended with an execution record that separates the model’s proposal, the harness decision, and the service result. That record is also the evidence needed to evaluate the system.
The final answer alone cannot show whether the agent repeated searches, lost a citation, exceeded its budget, or crossed a policy boundary. Two trajectories can produce similar prose while differing sharply in reliability, cost, and risk.
Agentic evaluation therefore asks two connected questions. Did the system produce a useful, supported result? Did it reach that result through an acceptable path?
Evaluate steps, trajectories, and outcomes
Evaluation operates at three nested units. Each reveals a different failure location.
| Unit | Core question | Example measures |
|---|---|---|
| Step | Was this action valid and useful? | tool selection, arguments, retrieval relevance, policy, latency |
| Trajectory | Did the sequence make safe and efficient progress? | completion, duplicate work, retries, handoff loss, cost |
| Outcome | Did the result help the user and remain supported? | task success, factuality, citation support, preference fit |
A step test can catch an invalid tool argument before the whole request fails. A trajectory test can catch twelve calls that should have taken four. An outcome test can catch a concise, orderly path that still recommends the wrong film.
No single score replaces these units. The evaluator should preserve the underlying judgments so a regression can be traced to retrieval, planning, state, policy, synthesis, or resource use.
The instrument below holds two trajectories fixed. Change the evaluation lens and watch the verdict move. Then expose the evidence available to the judge.
Same request · Similar final answer
Compare two films about identity, recommend the more ambiguous one, and add it to the shared watchlist only after approval.
Choose what counts
Passing threshold · 80
Score the final recommendation and ignore how it was produced.
Both paths pass
Score comparison
Outcome 100% · Path 0% · Safety 0% · Resources 0%
The answer-only lens passes both paths because their recommendations are nearly identical. A balanced rubric rejects Trajectory B for repeated work, lost evidence, and an unauthorized write attempt. Safety weighting makes that gap larger.
The exercise does not prove that one set of weights is universally correct. Evaluation criteria are product decisions: a system cannot be called reliable until its team states which outcomes, paths, risks, and resource limits count.
Compare paths that produce the same answer
Trajectory A completes both search branches, preserves the film records, applies the ambiguity preference, and stops after four calls. Its answer is supported by the work recorded beneath it.
Trajectory B reaches similar prose after twelve calls. It queries an unavailable source, repeats two searches, loses one supporting passage, and proposes a shared-watchlist write before approval.
An answer-level factuality score may treat those outputs as equivalent. The trajectory reveals four separate defects: poor routing, wasted work, lost provenance, and a policy violation.
These defects should remain separate measures before they are aggregated. A weighted score is useful for release decisions, but a single number cannot tell an engineer which component to repair.
The event record from Chapters 9 through 11 supplies the raw material: tasks, tool calls, source IDs, state transitions, validation results, policy decisions, approvals, latency, and cost. Observability turns that record into inspectable traces, evaluation applies explicit judgments to the traces, and monitoring tracks those judgments and operational measures over live traffic.
Trace the critical path rather than one total timer. Record queue time, each retrieval and model call, validation, retries, and fan-out and join boundaries under one request identifier. For streamed output, time to first token measures when the response begins while total duration measures when the work completes. Parallel child durations should not be summed as though they ran sequentially; the slowest dependency on the critical path determines wall-clock latency.
Aggregate the median (p50) for the common path and the 95th or 99th percentile (p95 or p99) for the tail, then segment by route, model, prompt version, index version, cache state, query class, attempt count, and terminal status. Those dimensions turn “responses are slow” into a bounded diagnosis. They also reveal when a small-model route saves money on most requests but creates a retry-heavy tail.
Build judgments before dashboards
Metrics become meaningful only after the team defines acceptable behavior on representative tasks. Start with a small judgment set that contains ordinary successes and deliberate failure cases. For the film task, cases can include a supported recommendation, a missing search branch hidden by fluent prose, a correct answer after duplicate work, and a safe pause before a shared write.
Each case needs expected outcomes and path constraints. The expected outcome might require Eternal Sunshine of the Spotless Mind or another well-supported choice, while path constraints can require two verified records and forbid a write before approval.
Some constraints are deterministic. Tool schemas, call limits, required approvals, source presence, and citation identifiers can be checked in code. Semantic quality still needs judgments about relevance, support, and preference fit.
The judgment set should include acceptable alternatives. A strong evaluator should not reject a different film merely because it differs from one reference answer when its evidence satisfies the request.
Version the tasks, traces, rubrics, and evaluator models together. A changed score is interpretable only when the team can identify which part of the evaluation system changed.
Calibrate model judges against people
Language models can judge semantic properties that fixed rules cannot capture cheaply. They can compare a recommendation with the request, assess whether evidence supports a claim, or explain a preference mismatch.
The generator and judge may share the same blind spots. A fluent answer can persuade a model judge even when the trace lost a source. A judge can also penalize a valid alternative because its rubric overfits one reference.
Calibration compares model judgments with decisions from people who understand the task. Disagreements reveal vague criteria, missing evidence, unstable prompts, or cases that require deterministic checks.
The instrument’s second control demonstrates this boundary. The model judge passes the answer text. A human cannot verify citation support until the trace and evidence are available, and once exposed, the missing passage makes the failure concrete.
Microsoft Research’s Agent-Pex extracts behavioral rules from prompts and traces, checks argument validity, output compliance, and plan sufficiency, and can invert rules into adversarial tests. The project reports evaluation across more than 5,000 Tau2 traces. It is an evaluation tool and research project; its results do not establish that automated judging removes the need for human calibration.
Self-RAG, an earlier research example of a model deciding when to retrieve and critiquing its own passages and generations, carries the same limit: benchmark results do not make self-critique an independent ground truth.
Test changes offline and watch them online
Offline regression tests should run after changes to prompts, tools, models, routing, retrieval, policies, or state handling. They provide repeatable comparisons before live users encounter the new behavior.
For a prompt comparison, freeze the retrieved evidence and structured response contract. Evaluate each system-prompt and model combination against the same status, citation, support, safety, latency, and cost checks. Preserve prompt text and version with the result. This isolates generation behavior from retrieval drift and makes a routing threshold testable against the cases it sends to each model.
Uber’s Cart Assistant combines curated synthetic requests with anonymized production-derived edge cases. Deterministic checks cover item identity, required items, schemas, quantities, and constraint propagation, while a multimodal model judge calibrated with humans scores semantic and experiential quality. Engineers compare candidate and baseline traces, then decide whether a disagreement exposes a product defect or an evaluator defect.
Production monitoring answers a different question: how does the released system behave on current traffic? It should track completion, retries, policy pauses, unsupported claims, latency, token use, tool errors, and user corrections.
Bayer and Thoughtworks’ PRINCE system runs dataset evaluation after workflow, prompt, or model changes, and daily evaluation over live traffic, measuring answer and context quality while monitoring intermediate workflow stages. PRINCE also exposes tool use and citations to users. That visibility supports both trust and diagnosis because a poor result can be traced back to its retrieval and execution path.
Live traces should feed new regression cases after review rather than becoming automatic training labels, because production traffic contains ambiguous requests, policy-sensitive data, and outcomes shaped by the current system.
Evaluate coordination by contribution
Multi-agent designs use the same evaluation units. A delegated step should be valid, the combined trajectory should preserve evidence and constraints, and the outcome should justify the added coordination.
Count handoffs, duplicated searches, conflicting conclusions, lost citations, and context copied across authority boundaries, and compare them with a single-agent or fixed-workflow baseline on the same tasks. Specialization earns its cost when it improves evidence coverage, tool isolation, or task quality enough to offset extra latency and failure surface. Agent count is an architectural choice rather than an evaluation result.
Close the loop without hiding the path
Part I built the search system: corpus, query understanding, retrieval, ranking, grounded generation, and evaluation. Part II placed those components inside a bounded control loop where the agent plans work, routes tasks to sources, preserves verified state, recovers from failures, and proposes actions within enforced authority. Every one of those decisions becomes part of the trajectory.
Evaluation closes the loop by testing the result and the route that produced it. The trace makes failures attributable, judgment sets make quality repeatable, and calibrated evaluators with production monitoring show whether changes improve the system under real conditions.
The resulting agentic search system remains grounded in its data. Better models can improve planning and interpretation, but the agent can act only on the sources, tools, state, and policies the application connects.
That system still assumes one shared body of eligible evidence. When different principals may retrieve different records, authorization has to constrain the candidate set before ranking or generation can observe it. Chapter 13 opens Part III by following that decision through lexical and dense retrieval.