Combine Evaluator Types

Every evaluator has a failure mode

Deterministic checks validate JSON, required citations, forbidden strings, numeric support, length, and exact task rules quickly and repeatably. Human reviewers handle nuance and real usefulness but cost time and can disagree. An LLM-as-judge can scale semantic comparisons, yet carries model bias, prompt sensitivity, inconsistency, and potential preference for verbose or self-similar answers.

Calibrate a judge against a representative human-reviewed set. Measure agreement, inspect disagreements, hide model identity, randomize answer order in pairwise comparisons, and include duplicated cases to estimate consistency. Never let a judge model evaluate sensitive content through an unapproved external service.

layer 1: deterministic contract and safety checks
layer 2: calibrated pairwise judge for supported dimensions
layer 3: sampled human review and critical-case adjudication

Warning: A judge's numeric precision can create false authority. 8.3/10 is not objectively meaningful unless scoring anchors and calibration show what differences the evaluator can reliably distinguish.

Note: The existing automated RAG evaluation module deliberately uses lightweight deterministic checks because the offline tiny LLM is not a reliable universal judge.