Debugging Bad RAG Answers

Find the failing stage

Record the normalized query, retrieved IDs and scores, prompt-template version, model tag, latency (elapsed response time), and final answer. If the right chunk never appeared, repair ingestion or search. If it appeared but the model ignored it, repair the prompt or choose a more suitable model.

Common failures include an out-of-date index, mixed embedding models, poor chunk boundaries, missing metadata filters, requesting too few top-k results, cutting text off when the context is full, and a prompt that rewards guessing.

Tip: Debug retrieval before generation. Printing the top chunks is cheaper and more informative than repeatedly changing the LLM prompt.