RAG Failed — But Which Layer Actually Failed? A Diagnostic Method

A RAG system returns a weak, wrong, incomplete, or unsupported answer. The usual diagnosis is “retrieval failed” or “the model hallucinated.” Both labels are too broad to be useful. A production RAG pipeline can fail before retrieval, during retrieval, while ranking, while assembling context, during generation, or after generation when evidence and validity are checked.

Why “RAG failed” is not a diagnosis

Retrieval-augmented generation combines several mechanisms: a user request is interpreted, one or more searches are constructed, candidate material is retrieved, results are filtered or reranked, selected evidence is inserted into a model context, and a model generates an answer. Production systems may add permissions, metadata filters, freshness rules, citations, query rewriting, hybrid search, tool calls, memory, and external state.

A wrong final answer therefore does not tell you which component failed. The model may have received the wrong evidence. It may have received the right evidence mixed with too much noise. The evidence may be correct but stale. The source may never have contained the answer. Or the model may have ignored perfectly adequate context.

OpenAI's RAG guidance already makes a fundamental distinction between retrieval failure and model failure: a system can supply the wrong context, or it can supply the right context and still generate the wrong answer. AWS similarly separates retrieve-only evaluation from retrieve-and-generate evaluation. For production diagnosis, that distinction should be taken further.

The RAG Failure Stack

Layer 1 — Source coverage: can the system answer this at all?

Before tuning embeddings, rerankers, or prompts, verify that the answer exists in the knowledge space the system is allowed to use. This sounds obvious, but many RAG failures are actually corpus failures. The requested fact may be absent, hidden in an unindexed attachment, available only in a newer document, stored in a system outside the RAG corpus, or blocked by permissions.

A retrieval metric cannot recover information that was never indexed. A larger top-k cannot retrieve a document the pipeline does not contain. If the source coverage test fails, the correct fix is ingestion, source selection, permissions, or an explicit “not answerable from available evidence” behaviour.

Layer 2 — Query construction: did the system ask the corpus the right question?

The user query is not always the retrieval query. Production systems rewrite questions, resolve pronouns, extract entities, translate languages, add metadata constraints, split complex questions, or generate multiple searches. Every transformation can improve retrieval, but every transformation can also destroy information.

A request such as “Does the policy still apply to contractors in Germany after the September update?” contains at least an entity, a population, a jurisdiction, and a time boundary. A rewritten query that becomes “contractor policy” may retrieve semantically related text while losing the variables that decide whether the answer is valid.

Layer 3 — Candidate retrieval: did the relevant evidence enter the set?

Candidate retrieval is primarily a recall problem. The diagnostic question is not yet whether the best result ranked first; it is whether relevant evidence appeared anywhere in the candidate pool. If the known correct source does not appear, investigate indexing, chunking, embeddings, lexical matching, metadata, hybrid search, language handling, synonyms, and query expansion.

This is where retrieval-only evaluation is valuable. AWS exposes context relevance and context coverage for retrieve-only RAG evaluation. The important production habit is to evaluate retrieval before generation so that a polished final answer cannot hide a weak candidate set.

Layer 4 — Ranking and filtering: was the right evidence discarded or buried?

A system can have good recall and still fail because the relevant evidence ranks below noisy but semantically similar material. Rerankers, recency boosts, authority weights, language preferences, tenant filters, access controls, product status filters, and deduplication all change what survives into the final context.

Debugging should therefore preserve the full candidate list, not only the final top-k. If the gold evidence was retrieved at rank 18 and a reranker removed it, the fix is not the same as a retrieval miss.

Layer 5 — Context assembly: did useful evidence become usable context?

Retrieval success does not guarantee context success. Relevant chunks can be truncated, separated from their qualifiers, duplicated until they dominate the prompt, mixed with contradictory versions, or surrounded by enough irrelevant text that the decisive passage loses salience.

Chunk boundaries are especially important. A sentence may contain the rule while the following sentence contains the exception. If they are indexed separately and only the first is retrieved, the retriever can appear relevant while the assembled context becomes misleading.

Layer 6 — Generation: can the model use correct evidence correctly?

Once the system has demonstrably supplied sufficient evidence, generation becomes independently testable. The model may overgeneralize, combine incompatible passages, ignore a negative statement, fail to follow the requested answer format, invent a bridge between facts, or answer from parametric memory instead of the retrieved evidence.

This is why end-to-end correctness alone is insufficient for diagnosis. OpenAI recommends evaluation as a structured way to understand application behaviour, while Anthropic's agent-evaluation guidance emphasizes multiple trials, graders, traces, and realistic failure cases. For RAG, the generator should be tested both with normal retrieval and with controlled gold context.

Layer 7 — Evidence attribution: is the answer actually supported?

A plausible answer with citations can still be weakly grounded. The cited document may be relevant to the topic but not support the specific claim. One sentence may be supported while another is inferred. A citation may point to a source that contradicts the answer once its conditions are read.

Citation evaluation therefore belongs after generation. AWS distinguishes citation precision from citation coverage: whether cited passages are correctly cited and whether the answer is sufficiently supported by citations. In production, claim-level support is more useful than treating the presence of any citation as evidence quality.

Layer 8 — Validity and freshness: was the evidence correct for this version of reality?

RAG can retrieve a perfectly authentic, highly relevant, faithfully quoted source and still produce a wrong answer if the source is no longer valid for the current question. Policies change. APIs are deprecated. prices move. software behaviour changes between versions. product inventory changes. permissions change. game patches change mechanics.

This is a separate failure class from hallucination. The evidence is real; its applicability is wrong. A robust system therefore needs timestamps, version or jurisdiction metadata where relevant, source authority, supersession rules, and an explicit mechanism for deciding when older evidence must be restricted or abandoned.

The fastest isolation method: the oracle-context test

The most useful first split is simple: manually provide the generator with a small set of evidence that you know is sufficient to answer the question. Keep the task and expected answer unchanged.

A production diagnostic sequence

Do not change three layers at once

A common debugging mistake is to change embeddings, chunk sizes, top-k, prompts, and the model in one iteration. If the score improves, you do not know why. If it gets worse, you do not know which change caused the regression.

Treat RAG debugging like experimental diagnosis: hold as much of the pipeline constant as possible and replace one uncertain component with a controlled input. Gold documents isolate retrieval. Gold chunks isolate chunk selection. Fixed context isolates generation. A fixed model isolates retrieval changes. A fixed corpus isolates ingestion and indexing changes.

A failure matrix for common RAG symptoms

Measure each layer with the metric it can actually influence

A correct answer can still hide a RAG defect

The reverse problem also matters. A RAG system can produce the correct answer while retrieval is broken. The model may already know the answer from training, infer it from weak evidence, or guess correctly. If evaluation looks only at the final answer, the system can appear healthy until the question reaches information that exists only in the private corpus.

This is the same reliability problem that appears in agent systems more broadly: outcome correctness is not enough to prove that the execution path was reliable. For RAG, traces should preserve at least the retrieval query, candidate set, ranking, final context, answer, citations, model version, corpus/index version, and relevant filters.

Use competing hypotheses, not a favourite explanation

If a bad answer immediately becomes “an embedding problem,” the investigation is already biased. A stronger debugging method writes down competing hypotheses before changing the system: missing source, bad query rewrite, low retrieval recall, bad reranking, context truncation, conflicting versions, generation failure, citation failure, or stale evidence.

Then choose a test that would separate those hypotheses. This is more efficient than collecting more examples that support the first explanation. The same principle applies to AI-assisted technical reasoning in general: a useful diagnosis is one that survives discriminating tests, not one that merely sounds plausible.

What would change this answer?

The exact diagnostic layers change with architecture. A simple single-document RAG application may have no query rewriting, reranker, or citation layer. An agentic retrieval system may add planning, multiple searches, tool selection, memory, permissions, and iterative evidence gathering. A structured database lookup may not use chunks or embeddings at all.

The core method still holds: identify the components that can independently change the result, construct controlled tests that replace uncertain components with known-good inputs, and measure each component using evidence appropriate to that layer.

Limitations

Real failures are often coupled. A weak query can reduce recall, which changes reranking, which changes context, which increases generation variance. The oracle-context test is a diagnostic shortcut, not proof that one component is solely responsible. Evaluation datasets can also be unrepresentative, and model-based graders can introduce their own errors.

The proposed stack is therefore best used as an investigation structure: log the pipeline, isolate variables, reproduce failures, test competing explanations, and keep end-to-end evaluation after layer-level fixes.

Conclusion

“RAG failed” should be the beginning of the investigation, not the conclusion. A useful diagnosis identifies whether the system lacked the evidence, searched incorrectly, failed to retrieve it, ranked it badly, assembled unusable context, generated incorrectly, attributed claims poorly, or applied evidence outside its validity boundary.

The practical rule is simple: replace uncertainty with controlled evidence one layer at a time. Start with the oracle-context test. Separate retrieval-only evaluation from generation evaluation. Preserve the full trace. Then fix the component that actually failed instead of tuning the entire RAG stack by intuition.

FAQ

Glossary

Primary sources and further reading