Debugging Bad RAG Answers

Find the failing stage

When a RAG pipeline gives a bad answer, the most useful habit is recording enough detail at request time to reconstruct exactly what happened afterward: the normalized query text, the retrieved chunk IDs and their similarity scores, which version of the prompt template was used, the model tag, the request's latency (how long the whole thing took, end to end), and the final generated answer. With that record in hand, debugging a bad answer becomes a matter of walking backward through the pipeline rather than guessing.

If the right chunk never even showed up among the retrieved results, the bug is upstream - in ingestion or search, from the previous two modules. If the right chunk did show up but the model's answer ignored or misread it, the bug is downstream - in the prompt construction or possibly the model choice itself. Common concrete causes worth checking, roughly in the order they tend to occur: an out-of-date index that was never refreshed after documents changed, mismatched embedding models between what was used to index versus what's used to query (the model-contract module's warning, showing up again in practice), chunk boundaries that split relevant content awkwardly, missing metadata filters that let irrelevant documents compete for the top spots, requesting too few top-k results, context getting silently cut off because the full prompt didn't fit the context window, and a prompt phrased in a way that quietly rewards the model for guessing rather than admitting uncertainty.

Tip: Debug retrieval before generation, every time. Simply printing out the top retrieved chunks and their scores is far cheaper - and far more informative - than repeatedly rewriting the LLM prompt and hoping the answer improves, when the actual problem might be that the right document was never even fetched.