Test Retrieval Quality

Retrieval needs evaluation

It's easy to build a retrieval system that runs without ever confirming it actually finds the right thing. The standard way to check is to build a small golden set: a curated list of test questions, each one paired with the specific document ID (or IDs) that a good retrieval system should return for it. For each test question, you run the real search and check whether the expected document actually shows up somewhere within the first k results - this "first k results" idea is commonly called top-k, and k is a number you choose (top-3, top-5, and so on) based on how many results your application can realistically use.

A good golden set deliberately includes more than just easy, clearly-answerable questions: reworded versions of the same question (to catch a system that only works with exact vocabulary matches), genuinely unclear or ambiguous questions, and - importantly - a few questions that have no good answer in the collection at all, to confirm the system doesn't confidently return an irrelevant document when it should really return nothing useful.

Two more tools narrow results before or after the similarity ranking itself. Metadata filters restrict the set of candidate records before similarity scoring even happens - for example, filtering to only documents belonging to the current customer, so one customer's private document can never surface in a different customer's search results, no matter how similar the wording happens to be. A similarity threshold, applied after scoring, rejects any match whose score falls below a chosen cutoff - a way of saying "if nothing scores well enough, admit that rather than returning the best of a bad set." That cutoff should be chosen by testing against real examples from your golden set, not guessed at.

Goal: The embedding-mismatch lab starts with a broken index/query contract - exactly the kind of silent, same-dimension-but-wrong-model failure described in the previous section. Repair it and prove the expected runbook document is actually retrieved.