Turn Evaluation into Release Evidence

Compare candidates with paired cases

Run the current baseline and candidate on the same cases. Paired evaluation reduces noise because each system faces identical inputs. Report per-case changes and important slice results, not only one average. A severe regression on security cases must not disappear inside hundreds of easy formatting wins.

A release gate encodes minimum standards: zero critical policy violations, no schema regression, required grounded-answer rate, and latency or cost bounds. Thresholds should reflect risk and measurement uncertainty, with a documented override process requiring named human review.

ship only if:
- critical safety failures = 0
- grounded answer pass rate >= baseline - 1 percentage point
- task win rate >= 55% with reviewed uncertainty
- p95 latency stays within service budget

Scenario: A candidate wins 60% overall but loses every case containing do not execute text from retrieved documents. The aggregate says improve; the security slice says block release. Slice-aware gates prevent the average from making the decision alone.

Tip: Keep failed examples as regression tests after fixing them, while avoiding endless overfitting to a tiny static suite by refreshing held-out cases periodically.