Define Quality Before Measuring

Generation has multiple dimensions

A generative answer can be fluent yet false, correct yet irrelevant, safe yet unhelpful, or useful but formatted incorrectly. A rubric names separate quality dimensions and concrete scoring anchors. Common dimensions include task correctness, faithfulness to evidence, completeness, relevance, style, safety, and schema compliance.

Build an evaluation dataset from real task distributions, important rare cases, adversarial inputs, multilingual or formatting variation where relevant, and explicit unanswerable examples. Each case needs expected facts, approved sources, or a human-written scoring guide - not necessarily one exact reference sentence, because many answers can be valid.

case: expired certificate incident
must include: identify expiry; recommend renewal workflow
must not include: invented command output; automatic destructive action
format: valid JSON with source_ids

Analogy: Judging an essay with only spelling accuracy misses argument quality and factual correctness. A rubric separates dimensions so one polished surface cannot hide a critical failure.

Tip: Version cases, rubrics, prompt templates, retrieval state, and model settings together. A score is meaningful only relative to the exact evaluation contract.