From Scores to Release Gates

Make quality checks repeatable

Run the golden dataset whenever documents, chunking, embeddings, retrieval settings, prompt templates, or models change. Record per-example output, aggregate scores, configuration, and revision identifiers. Averages alone can hide one severe policy failure.

A release gate might require all critical questions to cite approved sources, zero unsupported policy numbers, and a minimum retrieval success rate. Compare a candidate against the current baseline and investigate regressions rather than tuning only to pass a threshold.

Heuristic evaluation is a deliberate choice for this constrained lab: it needs no network, no extra model memory, and gives deterministic evidence. In production, combine deterministic checks, a validated judge model, task-specific rules, and human review.

Note: This exercise teaches Ragas-like evaluation concepts without importing Ragas or pretending a 0.5-billion-parameter model is a dependable universal judge.

Goal: Repair retrieval and answer grounding until the evaluator passes for the right reasons: valid citation, supported number, and high token support.