Faithfulness and Hallucination

Does the evidence support the claim?

Faithfulness measures whether claims in an answer are supported by the supplied context. A hallucination is unsupported generated content presented as though it were grounded. An answer can be factually true in the wider world yet unfaithful to the provided evidence.

Simple deterministic checks can compare meaningful answer tokens with context tokens, verify that cited IDs were actually retrieved, and require every number in the answer to occur in evidence. These checks are limited by synonyms and reasoning, but they are fast, explainable, and stable.

Ragas is a framework for evaluating RAG systems with metrics including faithfulness and relevance. Many judge-based workflows call another capable LLM. A tiny local model may be inconsistent as a judge, making the lesson flaky and confusing.

Warning: Never treat one automated score as truth. Review sampled failures, test evaluator calibration against human labels, and keep thresholds tied to a specific risk.

Scenario: A retention answer says 90 days while its cited deployment runbook contains no retention rule. Number-support and citation checks expose the unsupported claim immediately.