Test Retrieval Quality

Retrieval needs evaluation

Create a small golden set: test questions paired with the document identifiers (IDs) that should answer them. For each question, check whether the right document appears in the first k results; this group is called top-k. Include reworded questions, unclear questions, and questions with no relevant answer.

Metadata filters narrow the allowed records before similarity ranking. For example, filter by customer or product so one customer's document cannot appear in another customer's results. A similarity threshold rejects matches below a chosen score. Choose that score from test examples, not a guess.

Goal: The embedding-mismatch lab starts with a broken index/query contract. Repair it and prove the expected runbook is retrieved.