Skip to content
Search Engineering

Diagnose an existing system

Diagnostic Runbook

Start with the failure you can observe. The runbook will route you to the part of the system that most likely owns the cause, then provide checks, measurements, and interventions.

Start with what you can observe

What is going wrong?

Describe the failure or choose its visible effect. You do not need to know which part of the system caused it.

102 symptoms

  1. Wrong or weak resultsWhat Is Relevance?

    No operational definition of relevance

    No one can state what a relevant result is for this system in terms a user would recognize; quality debates settle by anecdote or seniority.

  2. Untrustworthy evaluationWhat Is Relevance?

    Nothing to measure against

    No labeled query set exists anywhere in the system; retrieval and ranking changes ship without measurement against known-good results.

  3. Untrustworthy evaluationWhat Is Relevance?

    Measuring something easier than relevance

    The team optimizes and reports proxies (benchmark scores, engagement, latency, corpus quality) while relevance for actual users goes unmeasured.

  4. Latency, cost, or capacityWhat Is Relevance?

    Slow responses with no attributable stage

    Responses became slow, but the system records only one total duration. No stage timing, route, model, cache state, retry count, or time-to-first-token dimension can explain the regression.

  5. Wrong or weak resultsWhat Is Relevance?

    Relevance treated as an algorithm property

    Ranking is a single algorithm's output (BM25 or embedding cosine); recency, locality, popularity, and behavior never reach the score. Results are topically right and situationally wrong.

  6. Wrong or weak resultsWhat Is Relevance?

    The toolbox reduced to its newest shelf

    The organization equates AI-powered search with embeddings plus LLMs; classical-ML options (signals boosting, LTR, semantic knowledge graphs, query understanding) never appear on the roadmap.

  7. Wrong or weak resultsWhat Is Relevance?

    One context modeled, two missing

    The system models only one of the three contexts (content, domain, user): plain keyword search, a disconnected knowledge graph, or context-blind collaborative filtering.

  8. Wrong or weak resultsWhat Is Relevance?

    Relevance by accumulated exception

    Hundreds of synonym entries, pinned results, and per-complaint business rules accumulate unowned and untested; in practice the loudest stakeholder defines relevance.

  9. Untrustworthy evaluationWhat Is Relevance?

    The system never learns

    The system never learns from user behavior; the same query returns the same wrong results indefinitely and quality plateaued long ago.

  10. Missing resultsCorpus and Indexing

    Index and query analysis chains disagree

    Index-time and query-time analysis chains disagree; terms that verifiably exist in documents never match because they were tokenized or stemmed differently on each side.

  11. Missing resultsCorpus and Indexing

    Aggressive analysis destroys distinctions

    Over-aggressive stemming or stopword removal destroys distinctions at index time; exact-token queries (SKUs, names, citations) conflate or vanish, and the loss is unrecoverable at query time.

  12. Latency, cost, or capacityCorpus and Indexing

    Chunking multiplies the index

    Chunking multiplies indexed document count by 100x to 10,000x; vector count, memory, and ANN build cost balloon (50M docs x 2K chunks = 100B vectors).

  13. Stale or unstable resultsCorpus and Indexing

    Vectors from a different model than the queries

    Embedding model swapped or finetuned without re-embedding the corpus; query vectors and document vectors occupy different spaces and similarity scores are noise, often patchily by document age.

  14. Missing resultsCorpus and Indexing

    Quantized index with no recovery path

    Index stores only quantized vectors; the over-fetch-and-rerank recovery pattern has nothing to rescore against, capping recall at the compressed ceiling (~60% for binary) with re-embedding as the only exit.

  15. Wrong or weak resultsCorpus and Indexing

    One bit per dimension, too few dimensions

    Binary quantization applied to a too-small embedding model (~384 dims); vector retrieval quality drops below the plain BM25 baseline.

  16. Latency, cost, or capacityCorpus and Indexing

    Segment merges rebuild the graph

    Indexing throughput collapses during segment merges on Lucene-family engines with HNSW fields; the graph is rebuilt on merge while query latency stays healthy, so the backlog grows unseen.

  17. Wrong or weak resultsQuery Understanding

    The query is matched as if it were a known item

    The system matches the query's tokens or embedding as if the query were a known item. Canonical case: 'iPhone' returns an iPhone 3, most like the token, least like the intent.

  18. Stale or unstable resultsQuery Understanding

    One intent, many query strings, diluted statistics

    The same intent expressed in multiple phrasings ('diet soda', 'sugar-free soda', 'zero calorie soda') accumulates separate, diluted statistics; every model trained on per-query signals underperforms.

  19. Wrong or weak resultsQuery Understanding

    Head intent, rare phrasing, no signal

    A head intent phrased rarely receives none of the ranking signal, conversion data, or boosts its head-query equivalent enjoys, because signal lookup is exact-string.

  20. Wrong or weak resultsQuery Understanding

    Multi-sense queries return blended senses

    Multi-sense queries ('mixer', 'server') return interleaved results from disjoint intents; no disambiguation, no clarification dialogue; dense query embeddings average the senses.

  21. Wrong or weak resultsQuery Understanding

    Broad intents ranked as if narrow

    Broad queries ('shirts', 'clothing') are ranked as if the user had a narrow target: popularity suppressed, near-duplicate top results, no refinement prompts.

  22. Missing resultsQuery Understanding

    Satisfiable natural-language queries return nothing

    Multi-token natural-language queries return zero results even though the intent is satisfiable ('top kimchi near Charlotte' as a literal AND query).

  23. Wrong or weak resultsQuery Understanding

    Operator tokens stripped as noise

    Operator-like tokens ('near', 'top') are stripped as stop words, discarding location-filter or boost semantics the query understanding layer should have executed.

  24. Wrong or weak resultsQuery Understanding

    The synonym dial has no good setting

    Per-token synonym expansion has no good setting: aggressive drifts from intent, conservative adds no recall; the team cycles through thesaurus edits without converging.

  25. Wrong or weak resultsQuery Understanding

    Expansion terms come from the wrong distribution

    Query expansion introduces terms alien to the corpus (out-of-the-box SPLADE expanding 'lasagna' to la, zagna, hotel, festival); precision drops on expanded queries.

  26. Stale or unstable resultsQuery Understanding

    The entity library falls behind the corpus

    The hand-curated entity library falls behind the corpus; queries with newer vocabulary parse as generic terms and lose structured handling.

  27. Wrong or weak resultsQuery Understanding

    Positional query-tree rewrites misfire

    A semantic function rewrites the query tree positionally and consumes an adjacent token of the wrong type ('near' followed by a non-city), producing a broken or empty filter.

  28. Wrong or weak resultsQuery Understanding

    The cache serves the wrong meaning

    The embedding-keyed cache serves an interpretation for a query that means something else ('short heel shoes' vs 'long heel shoes' scoring above 0.95 under a generic encoder).

  29. Latency, cost, or capacityQuery Understanding

    LLM-in-the-loop on every query

    One or more LLM calls run on every query for rewriting, classification, or extraction, dominating query-path p95 latency and per-query cost, with no cache and no deterministic fast path.

  30. Untrustworthy evaluationRetrieval

    Confusion between two distinct recall measurements

    Quantization 'recall' numbers (e.g., 'binary quantization gives 60% recall') are measured against full-precision ranking rather than user-facing relevance. Confusing the two leads to wrong diagnoses.

  31. Wrong or weak resultsRetrieval

    Dense retrieval crowds out exact-term needs

    Pure dense or dense-heavy hybrid retrieval; exact-term queries (product IDs, names, technical tokens) systematically miss obvious matches; false positives elevated.

  32. Wrong or weak resultsRetrieval

    Naive fusion silently picks a winner

    Hybrid fusion produces erratic results because BM25 (unbounded) and cosine ([-1,1]) live on incompatible scales; one signal silently dominates without normalization.

  33. Missing resultsRetrieval

    Compression recall cap, missing over-fetch

    Quantized vector retrieval returns top-K directly. Recall caps at the quantization recall ceiling (~60% for binary). Team blames the embedding model when the fix is over-fetch + rerank.

  34. Latency, cost, or capacityRetrieval

    Lucene HNSW operational tax

    Indexing throughput drops sharply during segment merges on Lucene-family engines (Elasticsearch, OpenSearch, Solr). The HNSW graph is rebuilt; cost is proportional to graph density.

  35. Stale or unstable resultsRetrieval

    Cluster centroids age out

    IVF-indexed retrieval recall degrades over weeks/months on a corpus with significant new content. Cluster centroids become stale; new documents land in suboptimal clusters.

  36. Wrong or weak resultsRetrieval

    Selective filters destroy HNSW recall

    Adding selective filters (category, recency, user-scoped) to HNSW retrieval kills recall. Filters apply post-traversal; the graph walk doesn't know about them.

  37. Wrong or weak resultsRetrieval

    Reranker domain doesn't match corpus

    Cross-encoder fine-tuned on a different domain (e.g., MS Marco's general web QA) reranks poorly on the target corpus. Lab eval looks fine; production behavior is off.

  38. Stale or unstable resultsRetrieval

    Semantic-only ranking ignores time

    Cross-encoder ranks older content over newer equivalents. Users report stale results. Cross-encoders score semantic relevance only; recency is a non-semantic signal.

  39. Missing resultsRetrieval

    Answers split across chunks

    Relevant passages exist in the corpus but never get retrieved. Test queries with known correct chunks return surrounding-but-wrong chunks because answer spans cross chunk boundaries.

  40. Wrong or weak resultsRetrieval

    Unbounded scores break term expansion

    Using Elasticsearch/OpenSearch significant-text aggregations or similar unbounded-score primitives for query expansion. Can't threshold reliably across queries because scores don't normalize.

  41. Untrustworthy evaluationRanking and Fusion

    The ranker trains on its own output

    Click-trained ranking keeps promoting the same class of results; new or differently-shaped candidates never gain ground; offline metrics look fine because they are computed on the same biased labels.

  42. Untrustworthy evaluationRanking and Fusion

    Judgments inherit the legacy ranker's ordering

    Click-derived judgments strongly correlate with the legacy ranker's ordering. Items ranked high by the old system look relevant regardless of merit; CTR was computed over raw impressions.

  43. Untrustworthy evaluationRanking and Fusion

    Raw ratios carry no confidence

    Rarely-shown documents with one or two lucky clicks outrank consistently-validated ones. Judgment grades of 1.0 sit on a denominator of 1; no prior, no confidence weighting.

  44. Wrong or weak resultsRanking and Fusion

    The model rediscovers its own label source

    Feature importance shows popularity dominant and semantic similarity negligible. The model has rediscovered the proxy that generated its own labels.

  45. Untrustworthy evaluationRanking and Fusion

    Features computed at training time rather than click time

    Model performs well offline, poorly online; the gap concentrates in queries where volatile features (price, availability, badges) changed between click time and training time.

  46. Untrustworthy evaluationRanking and Fusion

    Train/test split at the wrong level

    Suspiciously strong offline metrics that evaporate in A/B tests. Train/test split was done at the document level; query-dependent feature vectors leak across the split.

  47. Stale or unstable resultsRanking and Fusion

    One user moves a head query

    A product or document climbs a head query's results with signal traceable to one or a few users. No per-user deduplication in the signals aggregation.

  48. Stale or unstable resultsRanking and Fusion

    Query variants split their evidence

    Boost tables full of near-duplicate keys ('iPad', 'ipad', 'iPad 2'), each with weak counts; boosting underperforms on queries that are obviously popular. No query normalization in the GROUP BY.

  49. Stale or unstable resultsRanking and Fusion

    Yesterday's winners never age out

    Last season's winners still rank on time-dated queries; users see the old holiday schedule. No half-life decay, or a half-life wrong by orders of magnitude for the domain.

  50. Wrong or weak resultsRanking and Fusion

    All signals count the same

    Boost values dominated by abundant low-confidence clicks; purchases barely move rankings; negative signals absent entirely. All signal types counted equally.

  51. Wrong or weak resultsRanking and Fusion

    Boosting judged outside its scope

    Team reports 'signals boosting doesn't work'; measurement was on the whole query distribution, or the corpus is transient so per-document signal never accumulates. A structural scope limit rather than a tuning failure.

  52. Wrong or weak resultsRanking and Fusion

    Click data enters the score twice

    Signals boost and LTR model disagree; rankings oscillate or the model appears to fight manual boosts. The same click-derived information enters the final score twice.

  53. Wrong or weak resultsRanking and Fusion

    Phase windows chosen by copy-paste

    Relevant candidates sit just below the rerank window and never surface, or latency blows up because an expensive scorer runs over a set a cheaper phase should have narrowed first.

  54. Unsupported answersGeneration

    Fabrication despite grounding

    Answers contain specific claims (titles, numbers, names) absent from the retrieved context, delivered in a confident tone. Spot-checks of answers against their source passages fail.

  55. Unsupported answersGeneration

    Upstream failures blamed on the model

    Wrong or fabricated answers get attributed to the generation model when the retrieved candidate set never contained the needed information. Team iterates on prompts while the defect is upstream.

  56. Unsupported answersGeneration

    The generator obeys the wrong master

    The generator follows instructions embedded in user input or retrieved content, overriding intended behavior. No guardrail layer screens input or output.

  57. Unsupported answersGeneration

    No refusal path

    Zero-result or irrelevant retrievals still produce a fluent, confident answer. No refusal path, no query relaxation, no signal to the user.

  58. Unsupported answersGeneration

    The budget is exceeded silently

    Assembled prompt exceeds the model's context budget; retrieved passages or conversation history are silently truncated and the answer ignores the dropped material.

  59. Unsupported answersGeneration

    Prompt wording substitutes for evaluation

    System prompts change without a fixed evidence set, versioned rubric, or per-case comparison. Teams cannot tell whether a prompt improved grounding, refusal behavior, safety, latency, or cost.

  60. Untrustworthy evaluationEvaluation and Feedback

    Only clicks are logged

    Only clicks are logged; the result lists users actually saw exist nowhere. Skips are invisible and position-bias correction is impossible. Unrecoverable retroactively.

  61. Untrustworthy evaluationEvaluation and Feedback

    Signals keyed on query text

    Signals key on query text (or a hash of it) rather than a per-instance ID; sessions with different result orderings collapse into one record, so actions attribute to results the user never saw.

  62. Untrustworthy evaluationEvaluation and Feedback

    Only the positive channel collected

    Only positive actions are collected; skips, returns, and remove-from-carts never subtract, so rejected results keep whatever boost they earned indefinitely.

  63. Untrustworthy evaluationEvaluation and Feedback

    Sessions cannot be reconstructed

    Raw signals lack the identifiers (query_id, user_id, timestamps) that post-processing needs to reconstruct sessions; session-scoped judgments and skip mining cannot be computed.

  64. Untrustworthy evaluationEvaluation and Feedback

    The click model drops sessions without clicks

    The click model silently drops every zero-click session; where zero-click share is high, most of the traffic contributes no evidence at all.

  65. Untrustworthy evaluationEvaluation and Feedback

    Judgments are raw CTR

    Judgments are raw CTR over impressions; per-rank CTR decays in a power law regardless of content, so whatever the legacy ranker put on top looks relevant.

  66. Untrustworthy evaluationEvaluation and Feedback

    One click, grade 1.0

    Tail (query, doc) pairs with one click on one examine carry grade 1.0 and outrank pairs backed by hundredfold evidence; no prior, no shrinkage.

  67. Untrustworthy evaluationEvaluation and Feedback

    The model learns the ranker that fed it

    The trained ranker reproduces the ranker that generated its training data; whole feature-space regions have zero examples and popularity compounds. Position-bias corrections do not reach this.

  68. Untrustworthy evaluationEvaluation and Feedback

    Labels treated as ground truth

    Click-model output flows straight into model training; label bugs surface weeks later as inexplicable trained-model behavior instead of failing a cheap reversible test.

  69. Untrustworthy evaluationEvaluation and Feedback

    The window outlives the cause

    Judgments aggregated over a long window reward documents for reasons that no longer hold (was on sale, was in stock, was in the news); aggregation erased the timestamp of the click's cause.

  70. Untrustworthy evaluationEvaluation and Feedback

    Conversions as labels import confounders

    Conversion-only labels favor cheap, easily purchased items and punish browsing-rich queries; search 'wins' experiments while downstream metrics sag, or the reverse.

  71. Untrustworthy evaluationEvaluation and Feedback

    The judgment key omits user state

    Per-query-string judgments conflate distinct intents in personalized systems; the effective query includes user state but the judgment key does not.

  72. Agent or tool failuresAgentic Orchestration

    Retries without convergence

    Many near-duplicate or semi-random query reformulations per session with no convergence; later queries in a session score no better than earlier ones. Signature of a missing judge-in-the-loop.

  73. Agent or tool failuresAgentic Orchestration

    Self-certified completion

    The agent declares success and returns confidently regardless of result quality; users see polished prose wrapped around poor results. Signature of a missing outer harness.

  74. Agent or tool failuresAgentic Orchestration

    Missing or theatrical loop bounds

    Sessions loop until an infrastructure timeout, or every session terminates at the max-iteration cap. Token spend per session has a long tail an order of magnitude above the median.

  75. Agent or tool failuresAgentic Orchestration

    The judge rubber-stamps

    The loop runs, the judge scores, the harness passes outputs, and users still report poor results. Judge-human agreement was never measured on this corpus.

  76. Agent or tool failuresAgentic Orchestration

    The loop pays for an upstream problem

    Sessions succeed only after many retries; tokens per successful session are high and rising with corpus complexity. The loop is compensating for weak retrieval or ranking tools.

  77. Agent or tool failuresAgentic Orchestration

    The tool hands the agent raw infrastructure

    The agent constructs malformed or subtly wrong backend queries: bad DSL nesting, invented field names, filters that silently match nothing. The tool hands the agent raw infrastructure.

  78. Agent or tool failuresAgentic Orchestration

    The tool fights the training distribution

    The agent uses a tool awkwardly or avoids it entirely; the tool's signature has no analog in the web-search-shaped tool-use training distribution.

  79. Agent or tool failuresAgentic Orchestration

    The agent parses its own tools

    The agent misreads its own tool results: wrong ids passed to follow-up calls, scores misattributed, results conflated across calls. The tool returns free text instead of structured fields.

  80. Agent or tool failuresAgentic Orchestration

    Mid-list options disappear

    Selection frequency by list position is U-shaped: the agent picks options from the start and end of a long in-prompt list and forgets the middle. Degrades after a couple hundred entries.

  81. Agent or tool failuresAgentic Orchestration

    One model for every job

    Every LLM call site (classification, judging, extraction, the loop) runs the most expensive model. Cost and latency are high; quality is no better than a mixed fleet would deliver.

  82. Agent or tool failuresAgentic Orchestration

    The route moves cost into failures

    A model router exists, but difficult requests fail or retry on the small route while routine requests still reach the large model. Route-specific quality, latency, and cost were never evaluated together.

  83. Agent or tool failuresAgentic Orchestration

    The loop recomputes cache-shaped work

    Routine head queries pay for a full agentic loop at query time. Latency is dominated by LLM round trips recomputing work a cache lookup would have produced.

  84. Access or policy failuresPermissioned Retrieval

    Policy runs after candidate generation

    Visible results are filtered, but counts, suggestions, features, or traces still reveal restricted records.

  85. Access or policy failuresPermissioned Retrieval

    Restricted records reach downstream stages

    A record outside the principal's scope reaches ranking or model context.

  86. Access or policy failuresPermissioned Retrieval

    Cache reuse ignores authorization class

    A cache entry is reused across authorization classes.

  87. Access or policy failuresPermissioned Retrieval

    Secure filtering damages eligible recall

    Filtered vector search is secure but misses useful records inside the eligible subset.

  88. Access or policy failuresPermissioned Retrieval

    A protected field already shaped a derivative

    A protected field is removed from output after it already influenced an embedding or feature.

  89. Access or policy failuresPermissioned Retrieval

    Audit data copies protected inputs

    Trace or audit data copies restricted content or raw identity claims.

  90. Access or policy failuresPolicy Through the Trajectory

    Correlation is mistaken for authority

    Unsigned trace baggage is treated as proof of identity, scope, or approval.

  91. Access or policy failuresPolicy Through the Trajectory

    An earlier permit becomes a transitive grant

    A downstream service accepts an earlier permit without making a current decision.

  92. Access or policy failuresPolicy Through the Trajectory

    Delegation targets the wrong receiver

    A delegated credential is replayed at a service it was not issued to.

  93. Access or policy failuresPolicy Through the Trajectory

    Checkpoint state outlives policy

    Checkpointed evidence resumes under an expired credential or older policy.

  94. Access or policy failuresPolicy Through the Trajectory

    Memory enters another authorization class

    Memory created under one principal or purpose enters another trajectory.

  95. Access or policy failuresPolicy Through the Trajectory

    The applied rule cannot be reconstructed

    A decision cannot be reproduced because its policy version was not recorded.

  96. Connector failuresConnectors and Trust Boundaries

    Indexed authorization data is stale

    Indexed authorization attributes lag or disagree with their source.

  97. Connector failuresConnectors and Trust Boundaries

    Identity context expired before use

    A connector supplies an expired or revoked identity assertion.

  98. Connector failuresConnectors and Trust Boundaries

    Transport succeeds after meaning changes

    A connector response still parses but no longer means what the consumer expects.

  99. Connector failuresConnectors and Trust Boundaries

    A required source fails open

    A missing required source silently becomes a broader or unsupported completion.

  100. Connector failuresConnectors and Trust Boundaries

    Throttling creates more traffic

    Parallel branches and retries multiply load after throttling.

  101. Connector failuresConnectors and Trust Boundaries

    An unknown effect is repeated

    A timed-out action is repeated before its original commit status is known.

  102. Connector failuresConnectors and Trust Boundaries

    An assertion has no attributable source

    A policy-relevant assertion cannot be traced to a source version and observation time.

Explore the system

Browse by system area

Use this path when you already know which part of the search system you need to inspect.

  1. What Is Relevance?Define what useful results mean before tuning the system that produces them.9 symptoms
  2. Corpus and IndexingFind where ingestion, analysis, chunking, and vector storage hide recoverable evidence.7 symptoms
  3. Query UnderstandingTrace how a short request becomes intent, structure, expansion, and executable search.13 symptoms
  4. RetrievalCompare lexical and vector retrieval, then locate where candidate generation fails.11 symptoms
  5. Ranking and FusionFind where labels, behavioral signals, and reranking stages distort the final order.13 symptoms
  6. GenerationSeparate missing evidence, unsupported synthesis, unsafe instructions, and context limits.6 symptoms
  7. Evaluation and FeedbackAudit whether observations, judgments, experiments, and outcomes support trustworthy learning.12 symptoms
  8. Agentic OrchestrationInspect loop control, evaluator quality, tool design, context pressure, and serving cost.12 symptoms
  9. Permissioned RetrievalLocate where restricted evidence enters candidates, derivatives, caches, or traces before policy can contain it.6 symptoms
  10. Policy Through the TrajectoryTrace current identity, target, scope, policy, and validity through branches, memory, caches, and resume.6 symptoms
  11. Connectors and Trust BoundariesClassify the external boundary, then test provenance, freshness, failure semantics, retries, and effects.7 symptoms