# Agentic Orchestration
Search Engineering · Diagnostic Runbook · Chapter 8
Interactive edition: https://mutapa.dev/search-engineering/reference/agentic-orchestration
Inspect loop control, evaluator quality, tool design, context pressure, and serving cost.
## Start with the common causes
1. [Retries have no improving signal](https://mutapa.dev/search-engineering/reference/agentic-orchestration.md#spaghetti_at_the_wall)
Reformulations multiply without raising result quality.
2. [The judge does not match human preference](https://mutapa.dev/search-engineering/reference/agentic-orchestration.md#evaluator_misalignment)
The loop reports success while users still reject the output.
3. [Retries compensate for weak retrieval tools](https://mutapa.dev/search-engineering/reference/agentic-orchestration.md#brute_force_masks_weak_tools)
Sessions succeed only after repeated searches and rising token cost.
## Browse all symptoms
### Loop control
- [Retries without convergence](https://mutapa.dev/search-engineering/reference/agentic-orchestration.md#spaghetti_at_the_wall): Many near-duplicate or semi-random query reformulations per session with no convergence; later queries in a session score no better than earlier ones. Signature of a missing judge-in-the-loop.
- [Self-certified completion](https://mutapa.dev/search-engineering/reference/agentic-orchestration.md#sycophancy_premature_done): The agent declares success and returns confidently regardless of result quality; users see polished prose wrapped around poor results. Signature of a missing outer harness.
- [Missing or theatrical loop bounds](https://mutapa.dev/search-engineering/reference/agentic-orchestration.md#runaway_iteration): Sessions loop until an infrastructure timeout, or every session terminates at the max-iteration cap. Token spend per session has a long tail an order of magnitude above the median.
- [The judge rubber-stamps](https://mutapa.dev/search-engineering/reference/agentic-orchestration.md#evaluator_misalignment): The loop runs, the judge scores, the harness passes outputs, and users still report poor results. Judge-human agreement was never measured on this corpus.
### Tool quality
- [The loop pays for an upstream problem](https://mutapa.dev/search-engineering/reference/agentic-orchestration.md#brute_force_masks_weak_tools): Sessions succeed only after many retries; tokens per successful session are high and rising with corpus complexity. The loop is compensating for weak retrieval or ranking tools.
- [The tool hands the agent raw infrastructure](https://mutapa.dev/search-engineering/reference/agentic-orchestration.md#raw_backend_exposed): The agent constructs malformed or subtly wrong backend queries: bad DSL nesting, invented field names, filters that silently match nothing. The tool hands the agent raw infrastructure.
### Tool interface
- [The tool fights the training distribution](https://mutapa.dev/search-engineering/reference/agentic-orchestration.md#off_distribution_tool_shape): The agent uses a tool awkwardly or avoids it entirely; the tool's signature has no analog in the web-search-shaped tool-use training distribution.
- [The agent parses its own tools](https://mutapa.dev/search-engineering/reference/agentic-orchestration.md#tool_returns_unstructured_prose): The agent misreads its own tool results: wrong ids passed to follow-up calls, scores misattributed, results conflated across calls. The tool returns free text instead of structured fields.
### Context and cost
- [Mid-list options disappear](https://mutapa.dev/search-engineering/reference/agentic-orchestration.md#context_rot_long_list): Selection frequency by list position is U-shaped: the agent picks options from the start and end of a long in-prompt list and forgets the middle. Degrades after a couple hundred entries.
- [One model for every job](https://mutapa.dev/search-engineering/reference/agentic-orchestration.md#frontier_model_overuse): Every LLM call site (classification, judging, extraction, the loop) runs the most expensive model. Cost and latency are high; quality is no better than a mixed fleet would deliver.
- [The loop recomputes cache-shaped work](https://mutapa.dev/search-engineering/reference/agentic-orchestration.md#agent_in_hot_query_path): Routine head queries pay for a full agentic loop at query time. Latency is dominated by LLM round trips recomputing work a cache lookup would have produced.
## Deeper inspection and interventions
## Concept
Agentic orchestration is the stage where the searcher itself becomes a program that can plan, act, observe, and retry. Instead of a fixed pipeline invoked once per query, an LLM agent issues queries against the rest of the system, reads what comes back, reasons about whether the results are good, and searches again differently if they are not. When a product manager asks for RAG, they are asking for an outcome: search that gives an LLM context. The agentic loop is the current implementation of that ask, and chunking plus a vector database is only one of several ways to satisfy it.
The stage exists to recover from the structural weakness of classic one-shot RAG: no chance for self-correction. The canonical motivating failure is the Paris query. A one-shot system asked about restaurants in Paris returns results for Paris, Texas, and nothing in the architecture can notice the mistake. An agent with reasoning and retry notices the mismatch and re-queries. Architecturally, this stage sits on top of the pipeline rather than at the end of it: a control layer that re-enters the pipeline as many times as it decides to, consuming query understanding, retrieval, ranking, and generation as tools rather than replacing them.
The canonical architecture has four separable parts, and most diagnosis at this stage reduces to checking which parts are present:
- **The inner loop (the agent):** a conversation loop between harness code and an LLM given tool specifications. Each round the model returns either a final response or a list of tool-call requests; the harness executes each requested function, appends the results, and calls the model again. The loop ends when the model stops requesting tools.
- **Tools:** plain functions whose name, docstring, and typed parameters are serialized into the prompt as JSON schema. Everything the agent knows about a tool, it knows from prose and schema. A load-bearing fact for tool design: LLM tool-use training data is heavily web-search-shaped, so tools that accept keywords and return ranked (title, description, score) tuples get better default behavior for free.
- **Judge-in-the-loop:** a second tool, typically named `score` or `evaluate`, that takes the query and a result list and returns a structured per-result quality signal. In practice it wraps whatever the team trusts: an LTR model, a cross-encoder, an LLM-as-judge sub-agent, or a retrieval-tuned small model. Adding the judge is the single largest qualitative jump in loop behavior: it converts retry from brute force into directed search.
- **The harness (the outer loop):** orchestration code around the agent that evaluates the agent's final output using signals the agent does not control and, when unsatisfied, feeds an automated user-style message back in ("those results were not useful, try harder") and reruns.
The judge and the harness are the most-confused distinction in the topic, and keeping them separate matters for every diagnosis below. The judge operates inside the inner loop, at the agent's discretion. The harness operates outside it, unconditionally. A system can have the first without the second, and its agent will still rubber-stamp its own work.
One heuristic governs budget decisions at this stage: the loop is part intelligence and part brute force. Intelligence lives in reasoning and, even more, in the tools (pre-computed relationships, embedding quality, trained rerankers, taxonomies). Brute force is the retry, paid for in tokens and latency on every query, forever. The two trade off directly, so invest in tool quality before loop sophistication.
**What good looks like at this stage:** the agent converges rather than flails. It issues a small number of well-chosen tool calls, uses an evaluation signal to decide when results are good, stops when they are, and escalates honestly when they cannot be made good. Iteration count and token spend stay modest because intelligence in the tools substitutes for brute force in the loop.
## Symptom catalog
The first two symptoms are the entry points for the whole chapter. They correspond to the two loop-closure checks (judge and harness), they are cheap to answer, and everything else in the catalog depends on them.
### `spaghetti_at_the_wall`: Retries without convergence
- **Observable signal:** The agent issues many near-duplicate or semi-random query reformulations per session with no visible convergence; sometimes it gives up, or asks the user something it could have determined itself. Later queries in a session score no better than earlier ones.
- **Frequency:** Very common. This is the default behavior of a reasoning model given a search tool and no feedback; the source course demonstrates the behavior in its own lab trace.
- **Cost of leaving unaddressed:** Token spend scales with retry count while quality stays flat. Sessions that could converge in two calls burn six to eight.
The structural cause: no evaluation signal inside the loop. Reasoning can generate query variations but has no way to learn which ones worked. Retry behavior is trained into reasoning models (their training data contains search, notice-the-mistake, reformulate trajectories), so the agent will retry regardless; without a judge the agent retries with enthusiasm but never converges. Quick test: count distinct tool queries per session and check whether later queries score better than earlier ones. Flat quality across retries is the signature. The fix is a judge tool (`evaluator_misalignment` covers building one that can be trusted).
### `sycophancy_premature_done`: Self-certified completion
- **Observable signal:** The agent declares success and returns confidently regardless of result quality. Users see polished, plausible prose wrapped around poor results.
- **Frequency:** Very common in systems that deployed an inner loop and stopped there.
- **Cost of leaving unaddressed:** Bad results ship with high apparent confidence, which erodes trust faster than visible failures would.
The structural cause: no outer harness. The agent's "done" is self-certified, and sycophantic models certify readily. The judge tool does not fix this, because the agent invokes the judge at its own discretion; the harness must apply signals the agent does not control (a reranker, LTR scores, labels, business rules). Coding agents call this primitive a hook: feedback sent automatically when the agent thinks it is done. The same mechanism applies to search agents. A companion guardrail: force the agent to return structured output (a ranked list with evaluator scores attached) rather than free prose, so the harness can inspect the self-evaluation instead of parsing a story about it. Quick test: run an external judge over a sample of "done" outputs and measure the fraction the harness would have rejected.
### `runaway_iteration`: Missing or theatrical loop bounds
- **Observable signal:** Sessions loop until an infrastructure timeout, or a cost graph with a long tail of sessions consuming an order of magnitude more tokens than the median. The complementary failure is the cap doing all the work: every session terminates at max iterations, meaning the agent never satisfies itself and the cap functions as the stopping criterion.
- **Frequency:** Common. Uncapped loops are common in first implementations; cap-as-stopping-criterion is common in second ones.
- **Cost of leaving unaddressed:** Unbounded worst-case cost and latency, or a cap that silently truncates every session at an arbitrary point.
The structural cause: missing or unmeasured iteration guards, and no satisfaction detection between the cap and the judge. Both loops need bounds: the inner loop bounded strictly (the source course's earlier treatment of agentic evaluation already prescribed a max-iteration count), the outer harness at two to three passes. More harness passes than that suggest fixing the inner loop or the tools rather than raising the cap. Quick test: histogram iterations per session. A spike at the cap or an unbounded tail both indicate the symptom.
### `evaluator_misalignment`: The judge rubber-stamps
- **Observable signal:** The loop runs, the judge scores, the harness passes outputs, and users still report poor results. Judge scores fail to track human judgments.
- **Frequency:** Common wherever a judge was deployed without an agreement measurement, which is most first deployments.
- **Cost of leaving unaddressed:** Every downstream mechanism built on the judge (harness pass rates, convergence curves, retry decisions) is theater. This symptom is the trust gate for the entire chapter: until judge-human agreement is measured, no other signal here is reliable.
The structural cause: judge-human agreement was never measured on this corpus. The methodology carries over intact from LLM-as-judge practice: sample judged results, collect human labels, compute agreement, prefer pairwise comparisons over rating scales. Expect 80% agreement to come easily and the last 5-10% to be the actual work. Small language models can hold the judge role if and only if that measurement passes; the standing caveat is that LLMs are strong on the language parts of relevance and weaker on images and numeric attributes, so a language-only judge scoring image-heavy or spec-heavy results will misalign by construction. Below roughly 80% agreement, stop trusting the loop's feedback until the judge is fixed.
One deployable trick belongs here because it changes judge effectiveness cheaply: frame evaluation as user satisfaction rather than technical relevance. The source course scores results with happy and unhappy faces rather than relevant/not-relevant labels, reasoning that a sycophantic model works harder to please the user than to satisfy an abstract relevance standard. The same trait that makes this framing effective makes self-certified completion worthless, which is why the harness exists.
### `brute_force_masks_weak_tools`: The loop pays for an upstream problem
- **Observable signal:** Sessions succeed, but only after many retries; token spend per successful session is high and rising with corpus complexity. Improving the agent's prompt or model does little.
- **Frequency:** Common, and commonly misdiagnosed as an agent problem, because the loop hides the upstream failure by paying for it.
- **Cost of leaving unaddressed:** A permanent retry tax on every query. In the source course's phrasing: with a complicated legal question and a dumb BM25 tool, the agent eats its context doing obvious searches over and over.
The root cause usually lives at Stage 3 (Retrieval) or Stage 4 (Ranking & Fusion), and only the bill is visible here. This is the intelligence-vs-brute-force tradeoff showing up as cost: retries substitute for tool quality, and retries are paid per query. Quick test, and the single most important routing instruction in this chapter: extract the agent's own logged tool queries, run them directly against the backend, and evaluate them with Stage 3 methods. If the tool's standalone quality is poor, fix the tool before touching the loop. A related provocation gets its answer here: agents can grep, so why build retrieval infrastructure at all? Because the token economics are bad for complex domains; the economical answer is good tools.
### `raw_backend_exposed`: The tool hands the agent raw infrastructure
- **Observable signal:** The agent constructs malformed or subtly wrong backend queries: bad DSL nesting, invented field names, filters that silently match nothing. Failures are diverse and hard to pattern-match.
- **Frequency:** Common in systems that wrapped an existing engine by exposing its native query language directly.
- **Cost of leaving unaddressed:** A long tail of unrelated-looking failures, each requiring the agent to get engine-specific semantics right from prose descriptions alone.
The structural cause: the closer the tool sits to raw infrastructure, the more syntax and domain decisions the agent can get wrong. Lift the abstraction level with typed, high-level parameters instead of a query DSL. Use enums to constrain values to the system's taxonomy and write docstrings as behavioral guidance. Quick test: read the tool's parameter schema. If it accepts raw engine syntax, the symptom is present by construction.
### `off_distribution_tool_shape`: The tool fights the training distribution
- **Observable signal:** The agent uses a tool awkwardly or avoids it: wrong parameter conventions, keyword-stuffing a structured API, ignoring the tool in favor of guessing.
- **Frequency:** Common for tools designed around a backend's convenience rather than the model's expectations.
- **Cost of leaving unaddressed:** Persistent prompt-engineering overhead to make the tool usable at all, and degraded default behavior on every call.
The structural cause: tool-use training data is dominated by web-search-shaped examples. Keywords in, ranked (title, description, score) tuples out matches that distribution and gets good default behavior for free; shapes without an analog in the distribution do not. Quick test: compare the tool's signature against keywords-in, ranked-titles-out. The further away, the more prompt engineering it needs. This is the cheapest, most mechanistic tool-design constraint available, so apply it first, and depart from it only after measuring that something else works better.
### `tool_returns_unstructured_prose`: The agent parses its own tools
- **Observable signal:** The agent misreads its own tool results: wrong ids passed to follow-up calls, scores misattributed, results conflated across calls.
- **Frequency:** Common where tools were adapted from human-facing endpoints that return rendered text.
- **Cost of leaving unaddressed:** Brittle multi-step sessions; errors compound as the conversation grows and earlier results blur together.
The structural cause: the tool returns free text, so the agent must parse prose to recover structure, and that parsing is unreliable inside a long conversation. The fix keeps both channels: structured fields (id, score) for reliable downstream handling, title and description prose alongside for the agent to reason over. Quick test: read the tool's return shape. Structured fields with prose alongside is the pattern; prose alone is the symptom.
### `context_rot_long_list`: Mid-list options disappear
- **Observable signal:** The agent, or a classification step feeding it, reliably picks options from the start and end of a long in-prompt list and forgets the middle. Quality degrades as the option list grows, well before the context limit.
- **Frequency:** Common wherever a full taxonomy, catalog slice, or category list is inlined into a prompt.
- **Cost of leaving unaddressed:** Systematic bias against mid-list options that no amount of prompt wording fixes.
The structural cause: LLMs lose mid-context material as context fills. The source course's estimate is that option lists degrade after a couple hundred entries even in million-token windows. Quick test: plot selection frequency by list position; a U-shape is the signature. Mitigations are Stage 2 techniques: shortlist to roughly ten candidates via facet aggregation, hypothetical-then-resolve, or hierarchical traversal, and let the agent call a tool again rather than flooding its context up front. The same logic sets the per-call result count: around ten results per tool call, with further calls preferred over larger returns.
### `frontier_model_overuse`: One model for every job
- **Observable signal:** Every sub-task in the system (classification, judging, extraction, the loop itself) runs on the most expensive available model. Cost and latency are high; quality is no better than a mixed fleet would deliver.
- **Frequency:** Common. Systems tend to standardize on the model that worked in the prototype.
- **Cost of leaving unaddressed:** A cost multiple on every high-volume call site, with no quality return.
The structural cause: no one asked which tasks a small or batch-priced model could do. Constrained classification against a shortlist is small-model territory; judging is small-model territory once the alignment measurement passes; batch APIs cut pre-computation cost by roughly 10x (a source-course figure; verify against current pricing). Quick test: list every LLM call site with its model and traffic volume. Any high-volume call doing constrained classification or scoring on a frontier model is a candidate for downgrade.
### `model_router_mistuned`: The route moves cost into failures
- **Observable signal:** Small-route requests have elevated schema failures, corrections, fallbacks, or user dissatisfaction, while large-route traffic still contains routine constrained work. Tail latency rises because failed small calls escalate after spending most of their budget.
- **Frequency:** Common once a first cost router is tuned on intuition or average benchmark scores.
- **Cost of leaving unaddressed:** Apparent savings are paid back through retries, support load, and poor answers. The large route remains overused at the same time.
The router should produce an inspectable decision containing task class, route, model, policy version, reason, token and latency budgets, fallback, and final outcome. Start with deterministic features such as task type, output constraints, input size, ambiguity, tool requirements, and prior validation failure. A model-based router must recover enough quality or cost to pay for its own call.
Evaluate the full matrix by request class and route: quality, schema validity, refusal accuracy, fallback rate, median latency, tail latency, tokens, and cost. Include the fallback path in the test. Tune thresholds from the paired cases where small and large candidates disagree, then canary the policy with a rollback condition.
### `agent_in_hot_query_path`: The loop recomputes cache-shaped work
- **Observable signal:** Routine head queries pay for a full agentic loop at query time. Latency is dominated by LLM round trips doing work that does not vary between sessions.
- **Frequency:** Common in systems that adopted the loop as the single path for all traffic.
- **Cost of leaving unaddressed:** Loop latency (multiples of pipeline latency) and loop cost applied to the traffic that least needs it.
The structural cause: no separation between what must be reasoned live and what can be pre-computed. The source course's default is the opposite of loop-everything: structured query understanding is pre-computed and cached (keyed on query string or query embedding), and the live loop is reserved for sessions that need it. The anecdotal quality gain from adding an agent (roughly 15%, sometimes closer to 30%, varying tremendously by dataset, from informal experiments rather than a benchmark) is real enough to pay for on hard sessions and wasteful on head queries a cache already answers. Quick test: for a sample of sessions, diff the agent's per-session work against what a cache lookup would have produced. High overlap means the loop is recomputing cache-shaped work. Caching mechanics live at Stage 2; the architecture decision lives here.
## How to inspect
Inspection at this stage splits cleanly in two: code patterns that reveal how the loop and tools were built, and trace measurements that reveal how the loop behaves. The loop is undiagnosable without traces; if per-session iterations, tool calls, tokens, and judge scores are not being captured, enable that before anything else in this section.
### By code pattern
| Pattern / location | What you're looking for | What suggests a problem |
|---|---|---|
| The main agent loop (`while True`, `for _ in range(...)` around LLM calls) | Iteration bounds on the inner loop | Uncapped `while` loop; or a cap with no logging of cap hits (`runaway_iteration`) |
| Harness code around the agent, hook or callback registrations | An outer evaluation using signals the agent does not control | No code path evaluates the agent's final output; the agent's "done" is terminal (`sycophancy_premature_done`) |
| Tool registry / tool spec definitions | A judge or score tool alongside the search tools | Search tools only, no evaluation tool (`spaghetti_at_the_wall`) |
| Judge prompt text | Satisfaction framing and structured output | "Relevant"/"not relevant" wording where please-the-user framing would work harder; free-text verdicts |
| Tool parameter schemas | Typed high-level parameters, enums for vocabularies | A `query_dsl`, `es_query`, or raw JSON body parameter (`raw_backend_exposed`); free-string category parameters where an enum exists |
| Tool signatures vs. web-search shape | Keywords in, ranked (title, description, score) tuples out | Structured-API shapes with no training-distribution analog (`off_distribution_tool_shape`) |
| Tool return values | Structured fields (id, score) with prose alongside | Rendered text or markdown only (`tool_returns_unstructured_prose`) |
| Tool docstrings | Behavioral hints ("basic keyword search, don't try anything fancy") | Empty or purely mechanical docstrings; the docstring is most of what the agent knows |
| Prompt construction for classification or option selection | Shortlisting before the prompt is built | A full taxonomy or catalog list inlined into the prompt (`context_rot_long_list`) |
| Model ids at each LLM call site | Model routing matched to task difficulty | Frontier model ids at every call site, including constrained classification and scoring (`frontier_model_overuse`) |
| Router decision record | Task features, policy version, reason, budget, fallback, and final model | A bare model id or opaque router output makes `model_router_mistuned` unattributable |
| Query-path entry point | Routing between cached, classic, and agentic paths | Every query enters the loop; no cache lookup keyed on query string or embedding (`agent_in_hot_query_path`) |
| Final-response handling | Structured output contract (ranked list plus evaluator scores) | The harness parses free prose to find out what the agent concluded |
### By measurement
| Measurement | How to compute | Healthy range / red-flag threshold |
|---|---|---|
| Iterations per session (histogram) | Count inner-loop rounds per session from traces | Most sessions well under the cap; a spike at the cap or an unbounded tail indicates `runaway_iteration` |
| Cap-hit rate | Fraction of sessions terminating at max iterations | Near zero is healthy; a high rate means the cap is the stopping criterion |
| Tokens per session (distribution) | Sum prompt + completion tokens across the session | Watch the tail; sessions at 10x the median indicate runaway or brute-force compensation |
| Score-by-iteration curve | Judge score of best result so far, per iteration, averaged over sessions | Rising curve is convergence; flat curve is `spaghetti_at_the_wall` |
| Judge-human agreement | Sample judged results, collect human labels, compute agreement (pairwise beats rating scales) | Below roughly 80%, stop trusting loop feedback (`evaluator_misalignment`); the last 5-10% of agreement is the actual work |
| External-judge rejection rate on "done" outputs | Run a judge the agent cannot invoke over a sample of self-certified outputs | A meaningful rejection rate with no harness in place quantifies `sycophancy_premature_done` |
| Harness rejection rate | Fraction of agent outputs the harness bounces back | Persistent high rejection means the inner loop or tools need fixing rather than more passes |
| Standalone tool quality | Replay the agent's logged tool queries against the backend; evaluate with Stage 3 metrics (recall@K, MRR against labeled queries) | Poor standalone quality routes the diagnosis upstream (`brute_force_masks_weak_tools`) |
| Retries-to-success vs. tool quality | Join session retry counts with standalone tool quality over time | Rising retries as corpus complexity grows, with flat tool quality, is the brute-force signature |
| Selection frequency by list position | For classification/selection steps, plot chosen-option position | U-shape indicates `context_rot_long_list`; degradation begins around a couple hundred options |
| Cache-overlap on agent work | Diff per-session agent work against what a cache lookup keyed on query string/embedding would return | High overlap indicates `agent_in_hot_query_path` |
| Cost by call site and model | Group LLM spend by call site, model id, and traffic | High-volume constrained tasks on frontier models indicate `frontier_model_overuse` |
| Quality, latency, and cost by route | Join router decisions to terminal outcome; compare per request class, including fallback | Small-route retries or large-route routine work indicate `model_router_mistuned`; averages across routes hide both |
## Diagnostic questions
Triage flow for an agent or engineer assessing this stage. The first two questions are the loop-closure checks; they are cheap, and every other answer in this chapter is interpreted differently depending on them. Question 3 is the trust gate. Question 4 routes cross-stage misdiagnosis, which is more likely here than anywhere else in the runbook because the loop hides upstream failures by paying for them.
1. **[code] Does the agent have an evaluation signal inside the loop?** Look for a judge or score tool in the tool registry. Branches:
- No judge tool: expect `spaghetti_at_the_wall`; verify with the score-by-iteration curve, then add a judge wrapping whatever the team already trusts (LTR model, cross-encoder, LLM-as-judge sub-agent).
- Judge present: continue; its trustworthiness is question 3.
2. **[code] Does anything outside the agent evaluate its final output?** Look for harness code that scores the agent's result with signals the agent does not control and can bounce it back. Branches:
- No harness: expect `sycophancy_premature_done`; measure the external-judge rejection rate before and after adding one.
- Harness present: check that it inspects structured output rather than parsing prose, and that its pass signal is independent of the agent's own judge calls.
3. **[interview] Has judge-human agreement been measured on this corpus, and what was the number?** Branches:
- Never measured: `evaluator_misalignment` until proven otherwise; every metric downstream of the judge is unreliable. Measure now: pairwise comparisons, human labels, agreement target roughly 80% before trusting, with the last 5-10% expected to be real work.
- Measured and passing: small-model judges become a legitimate cost lever; note the caveat on image and numeric-attribute relevance.
4. **[code] What do the agent's own logged tool queries score when run directly against the backend?** Replay them and evaluate with Stage 3 methods. Branches:
- Standalone quality poor: the diagnosis routes to `retrieval` or `ranking_and_fusion`; fix the tool before touching the loop (`brute_force_masks_weak_tools`).
- Standalone quality fine: the loop itself is the right place to keep looking.
5. **[code] What bounds both loops, and what do the histograms show?** Branches:
- No inner cap, or no cap-hit logging: `runaway_iteration`; bound strictly and alert on cap hits.
- Every session hits the cap: the cap is the stopping criterion; the judge signal or satisfaction detection is not functioning.
- Harness passes exceed 2-3: fix the inner loop or tools rather than raising the pass count.
6. **[code] What shape are the tools?** Read parameter schemas, return shapes, and docstrings. Branches:
- DSL or raw engine syntax accepted: `raw_backend_exposed`; lift the abstraction level, constrain vocabularies with enums.
- Signature far from keywords-in, ranked-tuples-out: `off_distribution_tool_shape`; reshape toward web-search form unless measurements justify the departure.
- Prose-only returns: `tool_returns_unstructured_prose`; return structured fields with prose alongside.
- Empty docstrings: write behavioral hints; the docstring is a steering surface.
7. **[interview] What work happens live in the loop that could be pre-computed?** Branches:
- Query understanding recomputed per session; head queries entering the loop: `agent_in_hot_query_path`; cache keyed on query string or embedding, reserve the loop for sessions that need it.
- Deterministic work (graph traversal, corpus iteration, filtering) repeated by the agent: move it inside the tool, where it consumes no agent context.
8. **[code] Which model runs at each call site, and at what volume?** Branches:
- Frontier models on high-volume constrained classification or scoring: `frontier_model_overuse`; downgrade candidates to small models (after question 3 passes, for judges) and move pre-computation to batch APIs.
9. **[code] What does the router record, and how does each route perform by request class?** Branches:
- No policy version, reason, input features, budget, fallback, or final outcome: instrument the decision before tuning it.
- Small-route failures and fallbacks erase savings: `model_router_mistuned`; tighten eligibility or improve the small-route contract.
- Routine constrained work still reaches the large route: widen eligibility only after paired evaluation passes.
## Interventions by likely cause
Ordered by frequency of cause. Cross-references to symptoms in other stages use `stage#symptom_id` format. Note the third row: it is the map's explicit routing instruction, and it should be checked before any loop-tuning intervention below it is attempted.
| Hypothesis | Supporting evidence | Intervention | Expected impact | Cost / risk |
|---|---|---|---|---|
| **No judge inside the loop** (`spaghetti_at_the_wall`) | No score/evaluate tool registered; flat score-by-iteration curve | Add a judge tool wrapping an existing trusted scorer (LTR, cross-encoder, LLM-as-judge sub-agent); frame its output as user satisfaction rather than relevance | The largest single qualitative jump: retry becomes directed search; the agent learns within the session how to search this corpus | Low to medium: the wrapper is cheap; the alignment measurement (next rows) is the real work |
| **No outer harness** (`sycophancy_premature_done`) | Agent's "done" is terminal; external judge rejects a meaningful fraction of outputs | Add a harness that scores final output with agent-independent signals and bounces failures back with user-style feedback; cap at 2-3 passes; require structured output | Self-certified bad results stop shipping; failure becomes visible and retryable | Low: orchestration code around an existing loop |
| **Weak upstream tools, loop compensating** (`brute_force_masks_weak_tools`; routes to `retrieval#*`, `ranking_and_fusion#*`) | High retries-to-success; logged tool queries score poorly standalone; prompt/model changes don't help | Run Stage 3 diagnostics on the tools first; improve retrieval/ranking quality before touching the loop | Retries drop; per-session token cost falls; intelligence in tools replaces brute force paid per query | Varies with the upstream fix; the diagnosis itself is cheap (replay logged queries) |
| **Judge never validated against humans** (`evaluator_misalignment`) | No agreement measurement exists; users report poor results despite passing scores | Measure judge-human agreement (pairwise); fix the judge until agreement clears roughly 80%; re-check when corpus or judge changes | Every loop signal becomes trustworthy; unblocks small-model judges as a cost lever | Medium: human labeling effort; no model choice removes it |
| **Loops unbounded or cap doing all the work** (`runaway_iteration`) | No cap, cap-hit spike, or long token tail | Bound inner loop strictly, alert on cap hits; cap harness at 2-3 passes; add satisfaction detection between judge signal and cap | Worst-case cost and latency become predictable; cap-hit alerts surface tool and judge failures early | Low: configuration and alerting |
| **Tool exposes raw backend semantics** (`raw_backend_exposed`) | Parameter schema accepts DSL/raw syntax; diverse malformed-query failures | Lift the abstraction: typed high-level parameters, enums for vocabularies, docstrings as behavioral hints | The class of engine-semantics failures disappears by construction | Medium: tool redesign, plus prompt/regression re-testing |
| **Tool shape off the training distribution** (`off_distribution_tool_shape`) | Signature far from keywords-in, ranked-tuples-out; agent avoids or misuses the tool | Reshape toward web-search form: keywords in, ranked (title, description, score) tuples out | Better default behavior with less prompt engineering, for free | Low to medium: interface change, downstream handling updates |
| **Tools return prose only** (`tool_returns_unstructured_prose`) | Return shape is rendered text; agent misreads ids/scores in multi-step sessions | Return structured fields (id, score) with title/description prose alongside | Multi-step reliability improves; follow-up calls stop corrupting | Low: serialization change |
| **Full option lists in prompts** (`context_rot_long_list`) | U-shaped selection-by-position; quality degrades past a couple hundred options | Shortlist to ~10 via Stage 2 techniques (facet aggregation, hypothetical-then-resolve, hierarchical traversal); keep top_k per tool call near 10 and let the agent call again | Mid-list bias eliminated; context stays small enough to reason over | Low to medium: shortlisting is Stage 2 work |
| **Agent in the hot path for head traffic** (`agent_in_hot_query_path`) | High cache-overlap on per-session work; head-query latency dominated by LLM round trips | Pre-compute and cache query understanding (keyed on query string or embedding, Stage 2 mechanics); route head traffic to cached/classic paths, reserve the loop for complex or failed sessions | Head-query latency returns to pipeline levels; loop spend concentrates where it pays | Medium: a router and cache discipline; freshness now a cache concern |
| **Frontier models everywhere** (`frontier_model_overuse`) | High-volume constrained classification/scoring on frontier model ids | Route constrained tasks to small models (judges only after alignment passes); move pre-computation to batch APIs (roughly 10x cheaper, verify current pricing) | Large cost reduction at flat quality | Low to medium: routing logic plus per-task quality checks |
| **Router thresholds do not match task difficulty** (`model_router_mistuned`) | Small route has high failure or fallback rates; large route still handles routine constrained work | Emit an inspectable decision; evaluate both candidates on paired cases by request class; tune deterministic thresholds; canary with rollback | Cost falls without moving quality loss into retries or user corrections | Medium: evaluation matrix, trace dimensions, and policy rollout |
| **Existing search stack rebuilt instead of wrapped** (design-level; prevents several symptoms above) | A mature search engine sits unused while a parallel chunk-and-embed pipeline was built for "RAG" | Expose the existing engine as agent tools instead of rebuilding; chunking into vectors is only one implementation of RAG | Faster path to quality; the mature engine's relevance work carries over as tool intelligence | Low relative to the rebuild it replaces |