From Scores to Release Gates

Make quality checks repeatable

A golden dataset and a faithfulness checker are only genuinely useful if they're actually run on a regular, predictable schedule - specifically, whenever anything that could change an answer's accuracy is modified: the underlying documents, the chunking strategy, the embedding model, retrieval settings, prompt templates, or the LLM itself. Each run should record the full per-example output (not just an aggregate score, so a specific failing example can be inspected later), the aggregate scores, the exact configuration used, and the relevant revision identifiers - the same discipline the MLflow module built earlier in this track, applied specifically to evaluation runs. It's worth repeating a point from the previous section here: averages alone can comfortably hide one severe, specific policy failure sitting among many otherwise-fine answers.

A release gate is a concrete, automated rule that decides whether a candidate change is allowed to ship, built directly on top of this evaluation data - for example, requiring that all critical-category questions correctly cite an approved source, that there are zero unsupported policy numbers anywhere in the run, and that overall retrieval success stays above some measured minimum rate. The healthiest way to use a gate like this is to compare a candidate change against your last known-good baseline and actively investigate any regression, rather than iterating purely by tweaking the system until it happens to clear the threshold - the latter approach optimizes for passing the specific test, not for genuine quality.

It's a deliberate choice for this constrained learning environment to rely on heuristic, deterministic evaluation rather than an LLM-as-judge approach: it needs no network access, no extra memory budget for a second model, and it produces evidence that's fully deterministic and explainable. In a real production system, the strongest approach usually combines several of these tools together: deterministic checks like the ones covered here, a properly validated and calibrated judge model, task-specific business rules, and periodic human review - none of them alone is sufficient on its own.

Note: This exercise teaches the concepts behind Ragas-style evaluation without actually importing the Ragas library, and without pretending that a 0.5-billion-parameter model could reliably serve as a universal judge of answer quality - a real limitation this platform is upfront about rather than glossing over.

Goal: Repair retrieval and answer grounding until the evaluator passes for genuinely the right reasons: a valid citation, a supported number, and high token-level support between the answer and its evidence - not because a threshold was loosened to let a bad answer through.