Evaluate Retrieval and Generation Separately

One answer, two possible failure stages

The RAG-pipeline module earlier in this track already introduced the idea of debugging retrieval separately from generation when something goes wrong. This module builds that ad-hoc debugging habit into a repeatable, automated evaluation practice. As a quick refresher: retrieval-augmented generation (RAG) retrieves source passages relevant to a question, adds them into the prompt as reference material, and asks an LLM to answer using that material. When a RAG answer comes back wrong, there are exactly two places the failure could have occurred: retrieval may have supplied the wrong evidence in the first place, or generation may have had the right evidence available and still ignored or misused it.

A retrieval test checks a narrower, more mechanical question: did the expected document actually appear somewhere in the top results returned by the search step? A generation test checks a different question: given the context that was actually retrieved, does the final answer line up with it? Saving the full record for each test case - the query, the retrieved IDs, the retrieved context text, the generated answer, and any citations - is what makes a failure genuinely reproducible later, rather than a one-time observation nobody can revisit.

The foundation for running these tests repeatably is a small golden dataset: a curated set of questions, each paired with the source document IDs that are approved to answer it, and ideally the specific facts expected in a good answer. A well-built golden dataset deliberately includes answerable questions, questions with genuinely no supporting document in the collection, paraphrased versions of the same question, and tempting distractors - documents that look superficially relevant but shouldn't actually be cited. Version this dataset alongside your code and prompt templates, exactly the way the MLflow module earlier in this track recommended tracking every input that could affect a result.

Analogy: A researcher can fail in one of two distinct ways: by opening the wrong book entirely, or by opening the right book and then misquoting it. Before rewriting the conclusion of a report, the sensible first step is always to check the bibliography - was the right source even consulted?

Goal: In the lab, you'll prove that the wrong policy document actually entered the answer-generation stage - a retrieval-side failure - before touching the answer text itself, which would be treating a symptom rather than the actual cause.