Stage 7: Agentic Orchestration

A diagnostic reference. Read while optimizing an existing retrieval-augmented system. Each section serves either a human diagnosing the system or an agent interrogating it.

Concept

Agentic orchestration is the stage where the searcher itself becomes a program that can plan, act, observe, and retry. Instead of a fixed pipeline invoked once per query, an LLM agent issues queries against the rest of the system, reads what comes back, reasons about whether the results are good, and searches again differently if they are not. When a product manager asks for RAG, they are asking for an outcome: search that gives an LLM context. The agentic loop is the current implementation of that ask, and chunking plus a vector database is only one of several ways to satisfy it.

The stage exists to recover from the structural weakness of classic one-shot RAG: no chance for self-correction. The canonical motivating failure is the Paris query. A one-shot system asked about restaurants in Paris returns results for Paris, Texas, and nothing in the architecture can notice the mistake. An agent with reasoning and retry notices the mismatch and re-queries. Architecturally, this stage sits on top of the pipeline rather than at the end of it: a control layer that re-enters the pipeline as many times as it decides to, consuming query understanding, retrieval, ranking, and generation as tools rather than replacing them.

The canonical architecture has four separable parts, and most diagnosis at this stage reduces to checking which parts are present:

• The inner loop (the agent): a conversation loop between harness code and an LLM given tool specifications. Each round the model returns either a final response or a list of tool-call requests; the harness executes each requested function, appends the results, and calls the model again. The loop ends when the model stops requesting tools.
• Tools: plain functions whose name, docstring, and typed parameters are serialized into the prompt as JSON schema. Everything the agent knows about a tool, it knows from prose and schema. A load-bearing fact for tool design: LLM tool-use training data is heavily web-search-shaped, so tools that accept keywords and return ranked (title, description, score) tuples get better default behavior for free.
• Judge-in-the-loop: a second tool, typically named score or evaluate, that takes the query and a result list and returns a structured per-result quality signal. In practice it wraps whatever the team trusts: an LTR model, a cross-encoder, an LLM-as-judge sub-agent, or a retrieval-tuned small model. Adding the judge is the single largest qualitative jump in loop behavior: it converts retry from brute force into directed search.
• The harness (the outer loop): orchestration code around the agent that evaluates the agent’s final output using signals the agent does not control and, when unsatisfied, feeds an automated user-style message back in (“those results were not useful, try harder”) and reruns.

The judge and the harness are the most-confused distinction in the topic, and keeping them separate matters for every diagnosis below. The judge operates inside the inner loop, at the agent’s discretion. The harness operates outside it, unconditionally. A system can have the first without the second, and its agent will still rubber-stamp its own work.

One heuristic governs budget decisions at this stage: the loop is part intelligence and part brute force. Intelligence lives in reasoning and, even more, in the tools (pre-computed relationships, embedding quality, trained rerankers, taxonomies). Brute force is the retry, paid for in tokens and latency on every query, forever. The two trade off directly, so invest in tool quality before loop sophistication.

What good looks like at this stage: the agent converges rather than flails. It issues a small number of well-chosen tool calls, uses an evaluation signal to decide when results are good, stops when they are, and escalates honestly when they cannot be made good. Iteration count and token spend stay modest because intelligence in the tools substitutes for brute force in the loop.

Symptom catalog

The first two symptoms are the entry points for the whole chapter. They correspond to the two loop-closure checks (judge and harness), they are cheap to answer, and everything else in the catalog depends on them.

spaghetti_at_the_wall: Retries without convergence

• Observable signal: The agent issues many near-duplicate or semi-random query reformulations per session with no visible convergence; sometimes it gives up, or asks the user something it could have determined itself. Later queries in a session score no better than earlier ones.
• Frequency: Very common. This is the default behavior of a reasoning model given a search tool and no feedback; the source course demonstrates the behavior in its own lab trace.
• Cost of leaving unaddressed: Token spend scales with retry count while quality stays flat. Sessions that could converge in two calls burn six to eight.

The structural cause: no evaluation signal inside the loop. Reasoning can generate query variations but has no way to learn which ones worked. Retry behavior is trained into reasoning models (their training data contains search, notice-the-mistake, reformulate trajectories), so the agent will retry regardless; without a judge the agent retries with enthusiasm but never converges. Quick test: count distinct tool queries per session and check whether later queries score better than earlier ones. Flat quality across retries is the signature. The fix is a judge tool (evaluator_misalignment covers building one that can be trusted).

sycophancy_premature_done: Self-certified completion

• Observable signal: The agent declares success and returns confidently regardless of result quality. Users see polished, plausible prose wrapped around poor results.
• Frequency: Very common in systems that deployed an inner loop and stopped there.
• Cost of leaving unaddressed: Bad results ship with high apparent confidence, which erodes trust faster than visible failures would.

The structural cause: no outer harness. The agent’s “done” is self-certified, and sycophantic models certify readily. The judge tool does not fix this, because the agent invokes the judge at its own discretion; the harness must apply signals the agent does not control (a reranker, LTR scores, labels, business rules). Coding agents call this primitive a hook: feedback sent automatically when the agent thinks it is done. The same mechanism applies to search agents. A companion guardrail: force the agent to return structured output (a ranked list with evaluator scores attached) rather than free prose, so the harness can inspect the self-evaluation instead of parsing a story about it. Quick test: run an external judge over a sample of “done” outputs and measure the fraction the harness would have rejected.

runaway_iteration: Missing or theatrical loop bounds

• Observable signal: Sessions loop until an infrastructure timeout, or a cost graph with a long tail of sessions consuming an order of magnitude more tokens than the median. The complementary failure is the cap doing all the work: every session terminates at max iterations, meaning the agent never satisfies itself and the cap functions as the stopping criterion.
• Frequency: Common. Uncapped loops are common in first implementations; cap-as-stopping-criterion is common in second ones.
• Cost of leaving unaddressed: Unbounded worst-case cost and latency, or a cap that silently truncates every session at an arbitrary point.

The structural cause: missing or unmeasured iteration guards, and no satisfaction detection between the cap and the judge. Both loops need bounds: the inner loop bounded strictly (the source course’s earlier treatment of agentic evaluation already prescribed a max-iteration count), the outer harness at two to three passes. More harness passes than that suggest fixing the inner loop or the tools rather than raising the cap. Quick test: histogram iterations per session. A spike at the cap or an unbounded tail both indicate the symptom.

evaluator_misalignment: The judge rubber-stamps

• Observable signal: The loop runs, the judge scores, the harness passes outputs, and users still report poor results. Judge scores fail to track human judgments.
• Frequency: Common wherever a judge was deployed without an agreement measurement, which is most first deployments.
• Cost of leaving unaddressed: Every downstream mechanism built on the judge (harness pass rates, convergence curves, retry decisions) is theater. This symptom is the trust gate for the entire chapter: until judge-human agreement is measured, no other signal here is reliable.

The structural cause: judge-human agreement was never measured on this corpus. The methodology carries over intact from LLM-as-judge practice: sample judged results, collect human labels, compute agreement, prefer pairwise comparisons over rating scales. Expect 80% agreement to come easily and the last 5-10% to be the actual work. Small language models can hold the judge role if and only if that measurement passes; the standing caveat is that LLMs are strong on the language parts of relevance and weaker on images and numeric attributes, so a language-only judge scoring image-heavy or spec-heavy results will misalign by construction. Below roughly 80% agreement, stop trusting the loop’s feedback until the judge is fixed.

One deployable trick belongs here because it changes judge effectiveness cheaply: frame evaluation as user satisfaction rather than technical relevance. The source course scores results with happy and unhappy faces rather than relevant/not-relevant labels, reasoning that a sycophantic model works harder to please the user than to satisfy an abstract relevance standard. The same trait that makes this framing effective makes self-certified completion worthless, which is why the harness exists.

brute_force_masks_weak_tools: The loop pays for an upstream problem

• Observable signal: Sessions succeed, but only after many retries; token spend per successful session is high and rising with corpus complexity. Improving the agent’s prompt or model does little.
• Frequency: Common, and commonly misdiagnosed as an agent problem, because the loop hides the upstream failure by paying for it.
• Cost of leaving unaddressed: A permanent retry tax on every query. In the source course’s phrasing: with a complicated legal question and a dumb BM25 tool, the agent eats its context doing obvious searches over and over.

The root cause usually lives at Stage 3 (Retrieval) or Stage 4 (Ranking & Fusion), and only the bill is visible here. This is the intelligence-vs-brute-force tradeoff showing up as cost: retries substitute for tool quality, and retries are paid per query. Quick test, and the single most important routing instruction in this chapter: extract the agent’s own logged tool queries, run them directly against the backend, and evaluate them with Stage 3 methods. If the tool’s standalone quality is poor, fix the tool before touching the loop. A related provocation gets its answer here: agents can grep, so why build retrieval infrastructure at all? Because the token economics are bad for complex domains; the economical answer is good tools.

raw_backend_exposed: The tool hands the agent raw infrastructure

• Observable signal: The agent constructs malformed or subtly wrong backend queries: bad DSL nesting, invented field names, filters that silently match nothing. Failures are diverse and hard to pattern-match.
• Frequency: Common in systems that wrapped an existing engine by exposing its native query language directly.
• Cost of leaving unaddressed: A long tail of unrelated-looking failures, each requiring the agent to get engine-specific semantics right from prose descriptions alone.

The structural cause: the closer the tool sits to raw infrastructure, the more syntax and domain decisions the agent can get wrong. Lift the abstraction level with typed, high-level parameters instead of a query DSL. Use enums to constrain values to the system’s taxonomy and write docstrings as behavioral guidance. Quick test: read the tool’s parameter schema. If it accepts raw engine syntax, the symptom is present by construction.

off_distribution_tool_shape: The tool fights the training distribution

• Observable signal: The agent uses a tool awkwardly or avoids it: wrong parameter conventions, keyword-stuffing a structured API, ignoring the tool in favor of guessing.
• Frequency: Common for tools designed around a backend’s convenience rather than the model’s expectations.
• Cost of leaving unaddressed: Persistent prompt-engineering overhead to make the tool usable at all, and degraded default behavior on every call.

The structural cause: tool-use training data is dominated by web-search-shaped examples. Keywords in, ranked (title, description, score) tuples out matches that distribution and gets good default behavior for free; shapes without an analog in the distribution do not. Quick test: compare the tool’s signature against keywords-in, ranked-titles-out. The further away, the more prompt engineering it needs. This is the cheapest, most mechanistic tool-design constraint available, so apply it first, and depart from it only after measuring that something else works better.

tool_returns_unstructured_prose: The agent parses its own tools

• Observable signal: The agent misreads its own tool results: wrong ids passed to follow-up calls, scores misattributed, results conflated across calls.
• Frequency: Common where tools were adapted from human-facing endpoints that return rendered text.
• Cost of leaving unaddressed: Brittle multi-step sessions; errors compound as the conversation grows and earlier results blur together.

The structural cause: the tool returns free text, so the agent must parse prose to recover structure, and that parsing is unreliable inside a long conversation. The fix keeps both channels: structured fields (id, score) for reliable downstream handling, title and description prose alongside for the agent to reason over. Quick test: read the tool’s return shape. Structured fields with prose alongside is the pattern; prose alone is the symptom.

context_rot_long_list: Mid-list options disappear

• Observable signal: The agent, or a classification step feeding it, reliably picks options from the start and end of a long in-prompt list and forgets the middle. Quality degrades as the option list grows, well before the context limit.
• Frequency: Common wherever a full taxonomy, catalog slice, or category list is inlined into a prompt.
• Cost of leaving unaddressed: Systematic bias against mid-list options that no amount of prompt wording fixes.

The structural cause: LLMs lose mid-context material as context fills. The source course’s estimate is that option lists degrade after a couple hundred entries even in million-token windows. Quick test: plot selection frequency by list position; a U-shape is the signature. Mitigations are Stage 2 techniques: shortlist to roughly ten candidates via facet aggregation, hypothetical-then-resolve, or hierarchical traversal, and let the agent call a tool again rather than flooding its context up front. The same logic sets the per-call result count: around ten results per tool call, with further calls preferred over larger returns.

frontier_model_overuse: One model for every job

• Observable signal: Every sub-task in the system (classification, judging, extraction, the loop itself) runs on the most expensive available model. Cost and latency are high; quality is no better than a mixed fleet would deliver.
• Frequency: Common. Systems tend to standardize on the model that worked in the prototype.
• Cost of leaving unaddressed: A cost multiple on every high-volume call site, with no quality return.

The structural cause: no one asked which tasks a small or batch-priced model could do. Constrained classification against a shortlist is small-model territory; judging is small-model territory once the alignment measurement passes; batch APIs cut pre-computation cost by roughly 10x (a source-course figure; verify against current pricing). Quick test: list every LLM call site with its model and traffic volume. Any high-volume call doing constrained classification or scoring on a frontier model is a candidate for downgrade.

model_router_mistuned: The route moves cost into failures

• Observable signal: Small-route requests have elevated schema failures, corrections, fallbacks, or user dissatisfaction, while large-route traffic still contains routine constrained work. Tail latency rises because failed small calls escalate after spending most of their budget.
• Frequency: Common once a first cost router is tuned on intuition or average benchmark scores.
• Cost of leaving unaddressed: Apparent savings are paid back through retries, support load, and poor answers. The large route remains overused at the same time.

The router should produce an inspectable decision containing task class, route, model, policy version, reason, token and latency budgets, fallback, and final outcome. Start with deterministic features such as task type, output constraints, input size, ambiguity, tool requirements, and prior validation failure. A model-based router must recover enough quality or cost to pay for its own call.

Evaluate the full matrix by request class and route: quality, schema validity, refusal accuracy, fallback rate, median latency, tail latency, tokens, and cost. Include the fallback path in the test. Tune thresholds from the paired cases where small and large candidates disagree, then canary the policy with a rollback condition.

agent_in_hot_query_path: The loop recomputes cache-shaped work

• Observable signal: Routine head queries pay for a full agentic loop at query time. Latency is dominated by LLM round trips doing work that does not vary between sessions.
• Frequency: Common in systems that adopted the loop as the single path for all traffic.
• Cost of leaving unaddressed: Loop latency (multiples of pipeline latency) and loop cost applied to the traffic that least needs it.

The structural cause: no separation between what must be reasoned live and what can be pre-computed. The source course’s default is the opposite of loop-everything: structured query understanding is pre-computed and cached (keyed on query string or query embedding), and the live loop is reserved for sessions that need it. The anecdotal quality gain from adding an agent (roughly 15%, sometimes closer to 30%, varying tremendously by dataset, from informal experiments rather than a benchmark) is real enough to pay for on hard sessions and wasteful on head queries a cache already answers. Quick test: for a sample of sessions, diff the agent’s per-session work against what a cache lookup would have produced. High overlap means the loop is recomputing cache-shaped work. Caching mechanics live at Stage 2; the architecture decision lives here.

How to inspect

Inspection at this stage splits cleanly in two: code patterns that reveal how the loop and tools were built, and trace measurements that reveal how the loop behaves. The loop is undiagnosable without traces; if per-session iterations, tool calls, tokens, and judge scores are not being captured, enable that before anything else in this section.

By code pattern

Pattern / location	What you’re looking for	What suggests a problem
The main agent loop (while True, for _ in range(...) around LLM calls)	Iteration bounds on the inner loop	Uncapped while loop; or a cap with no logging of cap hits (runaway_iteration)
Harness code around the agent, hook or callback registrations	An outer evaluation using signals the agent does not control	No code path evaluates the agent’s final output; the agent’s “done” is terminal (sycophancy_premature_done)
Tool registry / tool spec definitions	A judge or score tool alongside the search tools	Search tools only, no evaluation tool (spaghetti_at_the_wall)
Judge prompt text	Satisfaction framing and structured output	”Relevant”/“not relevant” wording where please-the-user framing would work harder; free-text verdicts
Tool parameter schemas	Typed high-level parameters, enums for vocabularies	A query_dsl, es_query, or raw JSON body parameter (raw_backend_exposed); free-string category parameters where an enum exists
Tool signatures vs. web-search shape	Keywords in, ranked (title, description, score) tuples out	Structured-API shapes with no training-distribution analog (off_distribution_tool_shape)
Tool return values	Structured fields (id, score) with prose alongside	Rendered text or markdown only (tool_returns_unstructured_prose)
Tool docstrings	Behavioral hints (“basic keyword search, don’t try anything fancy”)	Empty or purely mechanical docstrings; the docstring is most of what the agent knows
Prompt construction for classification or option selection	Shortlisting before the prompt is built	A full taxonomy or catalog list inlined into the prompt (context_rot_long_list)
Model ids at each LLM call site	Model routing matched to task difficulty	Frontier model ids at every call site, including constrained classification and scoring (frontier_model_overuse)
Router decision record	Task features, policy version, reason, budget, fallback, and final model	A bare model id or opaque router output makes model_router_mistuned unattributable
Query-path entry point	Routing between cached, classic, and agentic paths	Every query enters the loop; no cache lookup keyed on query string or embedding (agent_in_hot_query_path)
Final-response handling	Structured output contract (ranked list plus evaluator scores)	The harness parses free prose to find out what the agent concluded

By measurement

Measurement	How to compute	Healthy range / red-flag threshold
Iterations per session (histogram)	Count inner-loop rounds per session from traces	Most sessions well under the cap; a spike at the cap or an unbounded tail indicates runaway_iteration
Cap-hit rate	Fraction of sessions terminating at max iterations	Near zero is healthy; a high rate means the cap is the stopping criterion
Tokens per session (distribution)	Sum prompt + completion tokens across the session	Watch the tail; sessions at 10x the median indicate runaway or brute-force compensation
Score-by-iteration curve	Judge score of best result so far, per iteration, averaged over sessions	Rising curve is convergence; flat curve is spaghetti_at_the_wall
Judge-human agreement	Sample judged results, collect human labels, compute agreement (pairwise beats rating scales)	Below roughly 80%, stop trusting loop feedback (evaluator_misalignment); the last 5-10% of agreement is the actual work
External-judge rejection rate on “done” outputs	Run a judge the agent cannot invoke over a sample of self-certified outputs	A meaningful rejection rate with no harness in place quantifies sycophancy_premature_done
Harness rejection rate	Fraction of agent outputs the harness bounces back	Persistent high rejection means the inner loop or tools need fixing rather than more passes
Standalone tool quality	Replay the agent’s logged tool queries against the backend; evaluate with Stage 3 metrics (recall@K, MRR against labeled queries)	Poor standalone quality routes the diagnosis upstream (brute_force_masks_weak_tools)
Retries-to-success vs. tool quality	Join session retry counts with standalone tool quality over time	Rising retries as corpus complexity grows, with flat tool quality, is the brute-force signature
Selection frequency by list position	For classification/selection steps, plot chosen-option position	U-shape indicates context_rot_long_list; degradation begins around a couple hundred options
Cache-overlap on agent work	Diff per-session agent work against what a cache lookup keyed on query string/embedding would return	High overlap indicates agent_in_hot_query_path
Cost by call site and model	Group LLM spend by call site, model id, and traffic	High-volume constrained tasks on frontier models indicate frontier_model_overuse
Quality, latency, and cost by route	Join router decisions to terminal outcome; compare per request class, including fallback	Small-route retries or large-route routine work indicate model_router_mistuned; averages across routes hide both

Diagnostic questions

Triage flow for an agent or engineer assessing this stage. The first two questions are the loop-closure checks; they are cheap, and every other answer in this chapter is interpreted differently depending on them. Question 3 is the trust gate. Question 4 routes cross-stage misdiagnosis, which is more likely here than anywhere else in the runbook because the loop hides upstream failures by paying for them.

1. [code] Does the agent have an evaluation signal inside the loop? Look for a judge or score tool in the tool registry. Branches:

   • No judge tool: expect spaghetti_at_the_wall; verify with the score-by-iteration curve, then add a judge wrapping whatever the team already trusts (LTR model, cross-encoder, LLM-as-judge sub-agent).
   • Judge present: continue; its trustworthiness is question 3.

2. [code] Does anything outside the agent evaluate its final output? Look for harness code that scores the agent’s result with signals the agent does not control and can bounce it back. Branches:

   • No harness: expect sycophancy_premature_done; measure the external-judge rejection rate before and after adding one.
   • Harness present: check that it inspects structured output rather than parsing prose, and that its pass signal is independent of the agent’s own judge calls.

3. [interview] Has judge-human agreement been measured on this corpus, and what was the number? Branches:

   • Never measured: evaluator_misalignment until proven otherwise; every metric downstream of the judge is unreliable. Measure now: pairwise comparisons, human labels, agreement target roughly 80% before trusting, with the last 5-10% expected to be real work.
   • Measured and passing: small-model judges become a legitimate cost lever; note the caveat on image and numeric-attribute relevance.

4. [code] What do the agent’s own logged tool queries score when run directly against the backend? Replay them and evaluate with Stage 3 methods. Branches:

   • Standalone quality poor: the diagnosis routes to retrieval or ranking_and_fusion; fix the tool before touching the loop (brute_force_masks_weak_tools).
   • Standalone quality fine: the loop itself is the right place to keep looking.

5. [code] What bounds both loops, and what do the histograms show? Branches:

   • No inner cap, or no cap-hit logging: runaway_iteration; bound strictly and alert on cap hits.
   • Every session hits the cap: the cap is the stopping criterion; the judge signal or satisfaction detection is not functioning.
   • Harness passes exceed 2-3: fix the inner loop or tools rather than raising the pass count.

6. [code] What shape are the tools? Read parameter schemas, return shapes, and docstrings. Branches:

   • DSL or raw engine syntax accepted: raw_backend_exposed; lift the abstraction level, constrain vocabularies with enums.
   • Signature far from keywords-in, ranked-tuples-out: off_distribution_tool_shape; reshape toward web-search form unless measurements justify the departure.
   • Prose-only returns: tool_returns_unstructured_prose; return structured fields with prose alongside.
   • Empty docstrings: write behavioral hints; the docstring is a steering surface.

7. [interview] What work happens live in the loop that could be pre-computed? Branches:

   • Query understanding recomputed per session; head queries entering the loop: agent_in_hot_query_path; cache keyed on query string or embedding, reserve the loop for sessions that need it.
   • Deterministic work (graph traversal, corpus iteration, filtering) repeated by the agent: move it inside the tool, where it consumes no agent context.

8. [code] Which model runs at each call site, and at what volume? Branches:

   • Frontier models on high-volume constrained classification or scoring: frontier_model_overuse; downgrade candidates to small models (after question 3 passes, for judges) and move pre-computation to batch APIs.

9. [code] What does the router record, and how does each route perform by request class? Branches:

   • No policy version, reason, input features, budget, fallback, or final outcome: instrument the decision before tuning it.
   • Small-route failures and fallbacks erase savings: model_router_mistuned; tighten eligibility or improve the small-route contract.
   • Routine constrained work still reaches the large route: widen eligibility only after paired evaluation passes.

Interventions by likely cause

Ordered by frequency of cause. Cross-references to symptoms in other stages use stage#symptom_id format. Note the third row: it is the map’s explicit routing instruction, and it should be checked before any loop-tuning intervention below it is attempted.

Hypothesis	Supporting evidence	Intervention	Expected impact	Cost / risk
No judge inside the loop (spaghetti_at_the_wall)	No score/evaluate tool registered; flat score-by-iteration curve	Add a judge tool wrapping an existing trusted scorer (LTR, cross-encoder, LLM-as-judge sub-agent); frame its output as user satisfaction rather than relevance	The largest single qualitative jump: retry becomes directed search; the agent learns within the session how to search this corpus	Low to medium: the wrapper is cheap; the alignment measurement (next rows) is the real work
No outer harness (sycophancy_premature_done)	Agent’s “done” is terminal; external judge rejects a meaningful fraction of outputs	Add a harness that scores final output with agent-independent signals and bounces failures back with user-style feedback; cap at 2-3 passes; require structured output	Self-certified bad results stop shipping; failure becomes visible and retryable	Low: orchestration code around an existing loop
Weak upstream tools, loop compensating (brute_force_masks_weak_tools; routes to retrieval#*, ranking_and_fusion#*)	High retries-to-success; logged tool queries score poorly standalone; prompt/model changes don’t help	Run Stage 3 diagnostics on the tools first; improve retrieval/ranking quality before touching the loop	Retries drop; per-session token cost falls; intelligence in tools replaces brute force paid per query	Varies with the upstream fix; the diagnosis itself is cheap (replay logged queries)
Judge never validated against humans (evaluator_misalignment)	No agreement measurement exists; users report poor results despite passing scores	Measure judge-human agreement (pairwise); fix the judge until agreement clears roughly 80%; re-check when corpus or judge changes	Every loop signal becomes trustworthy; unblocks small-model judges as a cost lever	Medium: human labeling effort; no model choice removes it
Loops unbounded or cap doing all the work (runaway_iteration)	No cap, cap-hit spike, or long token tail	Bound inner loop strictly, alert on cap hits; cap harness at 2-3 passes; add satisfaction detection between judge signal and cap	Worst-case cost and latency become predictable; cap-hit alerts surface tool and judge failures early	Low: configuration and alerting
Tool exposes raw backend semantics (raw_backend_exposed)	Parameter schema accepts DSL/raw syntax; diverse malformed-query failures	Lift the abstraction: typed high-level parameters, enums for vocabularies, docstrings as behavioral hints	The class of engine-semantics failures disappears by construction	Medium: tool redesign, plus prompt/regression re-testing
Tool shape off the training distribution (off_distribution_tool_shape)	Signature far from keywords-in, ranked-tuples-out; agent avoids or misuses the tool	Reshape toward web-search form: keywords in, ranked (title, description, score) tuples out	Better default behavior with less prompt engineering, for free	Low to medium: interface change, downstream handling updates
Tools return prose only (tool_returns_unstructured_prose)	Return shape is rendered text; agent misreads ids/scores in multi-step sessions	Return structured fields (id, score) with title/description prose alongside	Multi-step reliability improves; follow-up calls stop corrupting	Low: serialization change
Full option lists in prompts (context_rot_long_list)	U-shaped selection-by-position; quality degrades past a couple hundred options	Shortlist to ~10 via Stage 2 techniques (facet aggregation, hypothetical-then-resolve, hierarchical traversal); keep top_k per tool call near 10 and let the agent call again	Mid-list bias eliminated; context stays small enough to reason over	Low to medium: shortlisting is Stage 2 work
Agent in the hot path for head traffic (agent_in_hot_query_path)	High cache-overlap on per-session work; head-query latency dominated by LLM round trips	Pre-compute and cache query understanding (keyed on query string or embedding, Stage 2 mechanics); route head traffic to cached/classic paths, reserve the loop for complex or failed sessions	Head-query latency returns to pipeline levels; loop spend concentrates where it pays	Medium: a router and cache discipline; freshness now a cache concern
Frontier models everywhere (frontier_model_overuse)	High-volume constrained classification/scoring on frontier model ids	Route constrained tasks to small models (judges only after alignment passes); move pre-computation to batch APIs (roughly 10x cheaper, verify current pricing)	Large cost reduction at flat quality	Low to medium: routing logic plus per-task quality checks
Router thresholds do not match task difficulty (model_router_mistuned)	Small route has high failure or fallback rates; large route still handles routine constrained work	Emit an inspectable decision; evaluate both candidates on paired cases by request class; tune deterministic thresholds; canary with rollback	Cost falls without moving quality loss into retries or user corrections	Medium: evaluation matrix, trace dimensions, and policy rollout
Existing search stack rebuilt instead of wrapped (design-level; prevents several symptoms above)	A mature search engine sits unused while a parallel chunk-and-embed pipeline was built for “RAG”	Expose the existing engine as agent tools instead of rebuilding; chunking into vectors is only one implementation of RAG	Faster path to quality; the mature engine’s relevance work carries over as tool intelligence	Low relative to the rebuild it replaces