Evaluate Retrieval and Generation Separately

One answer, two possible failure stages

Retrieval-augmented generation (RAG) retrieves source passages, adds them to a prompt, and asks an LLM to answer. If the answer is wrong, retrieval may have supplied the wrong evidence or generation may have ignored correct evidence.

A retrieval test can ask whether the expected document appears in the top results. A generation test compares the answer with the retrieved context. Saving query, retrieved IDs, context, answer, and citations makes the failure reproducible.

Build a small golden dataset of questions, approved source IDs, and expected facts. Include answerable questions, questions with no supporting document, paraphrases, and tempting distractors. Version the dataset with the code and prompt.

Analogy: A researcher can fail by opening the wrong book or by misquoting the right book. Check the bibliography before rewriting the conclusion.

Goal: In the lab, prove that the wrong policy document entered the answer stage before changing the answer text.