Stage 5: Generation

A diagnostic reference. Read while optimizing an existing retrieval-augmented system. Each section serves either a human diagnosing the system or an agent interrogating it.

Concept

The generation stage turns a candidate set into an answer. The final ordered candidates from ranking and fusion are assembled into a prompt alongside the user’s question, and an LLM produces a response grounded in that context. The reason this stage exists in its current form is hallucination: LLMs produce convincing but not necessarily truthful output, and retrieval supplies something true for the model to lean on. Grounding reduces fabrication; it does not eliminate it.

In classic RAG the stage is structurally one-shot. A first LLM call constructs the query, retrieval runs, and a second LLM call answers with the results in context. There is no self-correction: if the constructed query misses or retrieval comes back thin, the answer model grounds itself on bad candidates and answers anyway. Two consequences follow for diagnosis. First, a large share of apparent generation failures are upstream failures surfacing at the last stage, because this is the first stage a user actually sees. Second, any recovery behavior (query relaxation, refusal, retry) must be built explicitly; the pipeline provides none by default. The agentic variants covered at Stage 7 give the generator the agency to re-search bad context, at a cost in tokens and latency.

Prompt assembly mechanics (passage ordering, source labeling, citation markers) and faithfulness metrics for generated answers are less settled than the rest of this stage; treat guidance there as starting points to validate rather than established practice.

What good looks like at this stage: every claim in the answer is supported by the retrieved context, the system refuses or retries when the context cannot support an answer, and generation failures are distinguishable from retrieval failures in the logs.

Symptom catalog

The catalog is deliberately short; a longer list would be invention.

retrieval_failure_masquerading: Upstream failures blamed on the model

• Observable signal: Wrong or fabricated answers are filed as “the model hallucinated.” Inspection of the failing requests shows the retrieved candidate set never contained the needed information: chunk boundaries split the answer span, the constructed query missed, or recall was simply low. The team iterates on prompt wording and model choice with no improvement.
• Frequency: Very common. The one-shot pipeline answers regardless of context quality, so every upstream defect exits through this stage.
• Cost of leaving unaddressed: Optimization effort lands on the wrong stage indefinitely. The generation model gets swapped, the prompt gets rewritten, and recall stays broken.

This symptom is listed first because it changes how every other symptom is read. Before any generation-stage intervention, read the retrieved context for a sample of failing requests. If the answer is not in the context, the defect belongs to Stage 3 (see retrieval#chunk_boundary_loss, retrieval#vector_dominance) or Stage 2 rather than here.

unsupported_confident_claims: Fabrication despite grounding

• Observable signal: Answers include specific claims (titles, numbers, names, quotes) that appear nowhere in the retrieved context, delivered fluently and without hedging. The source course’s canonical demonstration: asked without search for an author’s BM25 blog posts, the model invents plausible titles the author never wrote.
• Frequency: Common, and lower than in earlier model generations. The source course’s assessment: much less of a problem than in 2022, but still real even with search-augmented models.
• Cost of leaving unaddressed: User trust erodes on exactly the queries where the system sounds most authoritative. In legal, medical, or commerce domains the cost is direct.

Distinguish this from retrieval_failure_masquerading by checking the context: this symptom is present only when the context did contain adequate support and the model departed from it, or embellished beyond it.

injected_instruction_compliance: The generator obeys the wrong master

• Observable signal: The generator follows instructions embedded in its inputs rather than its system prompt. A widely reported incident: a Chevrolet dealership chatbot was manipulated into agreeing to sell a car for one dollar, via user text instructing it to agree to anything and call it a legally binding offer.
• Frequency: Rare as a discovered incident, common as a latent exposure. Any deployed system whose inputs reach the prompt unscreened carries it.
• Cost of leaving unaddressed: Reputational and potentially contractual. Severity scales with what the chatbot is authorized to say or do.

That incident is injection through user input. Retrieved documents are a second injection vector: adversarial text in the corpus reaches the prompt through retrieval. Mitigations: a guardrail model run in parallel with the main flow (reject and replace the response if it trips), and fine-tuned safety classifiers such as Llamaguard, which classify against an explicit taxonomy. Constraining the generator to structured output limits the blast radius of a successful injection.

empty_context_confident_answer: No refusal path

• Observable signal: Retrieval returns zero or irrelevant results and the system produces a fluent answer anyway. Logs show empty or near-empty candidate sets paired with confident responses. There is no code path that relaxes the query, and none that says “nothing found.”
• Frequency: Common in first implementations, where the prompt template unconditionally demands an answer.
• Cost of leaving unaddressed: The system’s worst answers are indistinguishable from its best. Users learn they cannot tell the difference and stop trusting either.

The source course prescribes both halves of the fix. Before generation: classic query relaxation (AND to OR to semantic search, or dropping filters). At generation: accept that true zero-result situations exist and say so. A system that always answers is worse than one that sometimes reports finding nothing.

context_overflow_truncation: The budget is exceeded silently

• Observable signal: Answers ignore passages the team believes were provided. Token counts of assembled prompts approach or exceed the model’s window, especially in multi-turn sessions where conversation history accumulates. Assembly code concatenates strings without measuring tokens.
• Frequency: Common in multi-turn deployments; rare in single-shot pipelines with small top-K.
• Cost of leaving unaddressed: Grounding quality degrades invisibly. The passages most likely to be dropped are the ones appended last, regardless of relevance.

Context windows are large (hundreds of thousands of tokens as of the source course, and growing) but finite, and headroom should be preserved for follow-on conversation. A related effect from lesson 12: even within the window, models lose track of mid-context material as prompts grow (context rot), which argues for fewer, better passages over aggressive context stuffing.

prompt_change_unmeasured: Prompt wording substitutes for evaluation

• Observable signal: A prompt is approved because it reads more forcefully or fixes a few hand-picked examples. The comparison changed retrieval results, model, sampling, or rubric at the same time, and no per-case report survives.
• Frequency: Very common because system prompts are easy to edit and difficult to assess by inspection.
• Cost of leaving unaddressed: Regressions move between answer quality, refusal, citation support, safety, latency, and cost without attribution. Prompt complexity grows as old clauses remain after their original failure disappears.

Hold the question, retrieved evidence, structured output contract, and rubric fixed. Compare prompt and model candidates across supported, insufficient-evidence, and adversarial cases. Preserve prompt text and version with status, citation, support, safety, latency, cost, and per-case results. A framework can run the matrix, but it cannot choose the rubric or replace human calibration of semantic judges.

How to inspect

By code pattern

Pattern / location	What you’re looking for	What suggests a problem
Prompt template string; join(...) over retrieved docs	The assembly step: what is injected, with what instruction	Unconditional “answer using these results” with no branch for empty or weak context suggests empty_context_confident_answer
len(results) == 0, if not candidates near the generation call	A zero-result branch	Absence of any such branch; generation call is unconditional
Token counting (tiktoken, count_tokens) in assembly code	Whether the prompt is measured against the window	String concatenation with no token measurement suggests context_overflow_truncation
Conversation history handling (messages.append, session store)	How multi-turn history accumulates	Unbounded history growth with fixed top-K injection
Guardrail calls (llamaguard, moderation, safety classifier) around input and output	A screening layer	No guardrail anywhere suggests latent injected_instruction_compliance
The instruction text of the prompt	Whether the model is told it may decline	No “if the context does not answer the question, say so” style instruction
Logging around the generation call	Whether context and answer are stored together	Answer logged without its context makes retrieval_failure_masquerading undiagnosable after the fact
Prompt files and release checks	Versioned prompts evaluated against fixed evidence and assertions	Prompt text edited inline with no checked-in cases or result artifact: prompt_change_unmeasured

By measurement

Measurement	How to compute	Healthy range / red-flag threshold
Support rate (claims backed by context)	Sample answers; check each factual claim against the retrieved passages, manually or with an LLM judge whose agreement with humans has been measured	Domain-dependent; any systematic unsupported-claim pattern is a red flag
Answerable-from-context rate on failures	For a sample of user-reported wrong answers, label whether the context contained the correct answer	If most failures were unanswerable from context, the problem is upstream: retrieval_failure_masquerading
Zero-result behavior	Count requests with empty or below-threshold candidate sets; inspect what was returned	Confident prose on empty candidates is empty_context_confident_answer
Prompt token distribution	Log assembled prompt token counts per request, p50/p95/max	p95 near the model window, or any request over it, flags context_overflow_truncation
Guardrail trip rate	If a guardrail exists, log trips per thousand requests	Zero trips ever may mean the guardrail is miswired rather than that inputs are clean
Session-level answer quality	Attribute user signals (copied quote, highlight, follow-up referencing a source) to the retrieved chunk they point at; average across many sessions	Individual sessions are noisy; the aggregate trend is the metric. See Stage 6 for the full method
Prompt candidate matrix	Run every prompt and model candidate over the same cases; compare status, citation, support, safety, latency, and cost by case	Missing combinations or aggregate-only reporting indicates prompt_change_unmeasured

Diagnostic questions

1. [interview] When an answer is wrong, what does the team look at first? If the answer is “the prompt” or “the model,” and nobody reads the retrieved context, suspect retrieval_failure_masquerading before anything else. Force the habit: sample ten failures, read their contexts, label each answerable or not.

2. [code] What exactly goes into the prompt, and in what order? Establish the assembly pipeline: how many candidates, which fields, what instruction, where the question sits. This grounds every later question.

3. [code] What happens when retrieval returns nothing, or nothing good? Branches:

   • No branch exists, generation always runs: empty_context_confident_answer. Add relaxation before generation and a refusal path within it.
   • Relaxation exists: ask how many rungs and whether a true zero-result response is permitted at the end of the ladder.

4. [code] Is the assembled prompt measured in tokens, and what is the multi-turn history policy? Branches:

   • No measurement, unbounded history: suspect context_overflow_truncation; add token logging first, then a budget.
   • Measured and budgeted: ask how much headroom is reserved for follow-on turns.

5. [code] Is there a guardrail on input, on output, or neither? Branches:

   • Neither: latent injected_instruction_compliance. Weigh a parallel guardrail; the parallel pattern hides its latency.
   • Guardrail present: ask what taxonomy it was trained or configured against, and whether that taxonomy matches this deployment’s actual risks.

6. [interview] For answers that pass a spot-check today, has anyone verified claims against sources systematically? Branches:

   • Never: run a support-rate sample (see measurements). Unsupported claims with adequate context is unsupported_confident_claims.
   • An LLM judge does it: ask whether judge-human agreement was ever measured; unmeasured judges inherit the alignment caveat from Stage 6.

7. [interview] Is the one-shot ceiling the actual constraint? If failures cluster on queries needing reformulation (the constructed query misses intent, and a human retrying with different words succeeds), the fix is architectural rather than prompt-level: see Stage 7 for the agentic loop that gives the generator retry capability.

8. [code] What gates a system-prompt change? No fixed evidence, versioned rubric, or per-case report means prompt_change_unmeasured. Freeze retrieval inputs first, compare prompt and model candidates, and calibrate any semantic judge against human decisions.

Interventions by likely cause

Ordered by frequency of cause in real systems. Cross-references use stage#symptom_id format.

Hypothesis	Supporting evidence	Intervention	Expected impact	Cost / risk
Failures are upstream; generation is being tuned in vain (generation#retrieval_failure_masquerading)	Sampled failing requests show the answer absent from retrieved context	Stop generation-side iteration; route the diagnosis to Stage 3 (retrieval#chunk_boundary_loss, recall measurement) or Stage 2	Optimization effort lands on the real bottleneck	Low: measurement and triage discipline
No refusal or relaxation path for weak retrievals (generation#empty_context_confident_answer)	Unconditional generation call; empty candidate sets paired with confident answers in logs	Add query relaxation before generation; add an explicit instruction and code path for “the context does not answer this”; permit true zero-result responses	Worst answers become honest instead of confident	Low: local pipeline change
Model departs from adequate context (generation#unsupported_confident_claims)	Support-rate sample shows unsupported claims where context held the answer	Tighten the grounding instruction; reduce injected passage count to fight context rot; add a claim spot-check to evaluation	Fewer fabrications on well-served queries	Low to medium: prompt iteration plus evaluation work
Prompt changes are not controlled experiments (generation#prompt_change_unmeasured)	Inline prompt edits, hand-picked demos, no fixed evidence or per-case report	Check in prompt variants and representative cases; hold evidence and rubric fixed; compare status, support, safety, latency, and cost before release	Prompt efficacy becomes attributable and regressions become reviewable	Low to medium: fixture and rubric work plus evaluation calls
Prompt assembly exceeds the context budget (generation#context_overflow_truncation)	Prompt token p95 near the window; unbounded conversation history	Log token counts; set an assembly budget with headroom for follow-on turns; cap or summarize history; prefer fewer, better passages over more	Grounding stops degrading silently in long sessions	Low: assembly-code change
No screening between inputs and generator (generation#injected_instruction_compliance)	No guardrail layer; chatbot is publicly reachable	Add a guardrail model in parallel (reject and replace on trip); consider a taxonomy-trained classifier such as Llamaguard; constrain output to a structured shape	Blocks the known injection class; limits blast radius of the unknown	Medium: extra model call per request, tuning the taxonomy to the domain
One-shot architecture cannot recover from query misses (course framing; no single symptom)	Failures cluster on queries a human retry would fix; two-stage pipeline with no feedback	Adopt the agentic loop: generator with search tools, retry capability, iteration cap. See Stage 7	Recovers reformulation-class failures	High: latency and token cost rise; new failure modes arrive with agency