Preference Optimization and Evaluation

Preferred is not automatically correct

Preference data compares candidate answers or assigns quality ratings. Techniques such as reinforcement learning from human feedback (RLHF) and direct preference optimization (DPO) use that signal to make preferred responses more likely. The exact algorithms differ, but both depend on what reviewers reward.

Reviewers may prefer confident, long, agreeable prose even when it is less factual. Preference optimization can therefore improve conversational feel while worsening calibration or truthfulness. Detailed rubrics, diverse reviewers, disagreement analysis, and separate factual/safety tests are essential.

preference objective: make answer A more likely than B
factual objective: claims supported by evidence
safety objective: forbidden action never occurs
evaluate all separately

Scenario: Tuned answers receive higher style ratings because they sound decisive, but unsupported recommendations rise. A faithfulness gate catches the regression that preference wins concealed.

Note: Always compare adaptation with a prompt/RAG baseline under the same test set and total operating cost. Training complexity must earn measurable value.