Labels Are Measurements

The answer column can be wrong

A label is not absolute truth merely because it sits in y. It is a measurement produced by a process: a human review, later business outcome, sensor threshold, or existing policy. Label noise means some recorded labels are wrong. Label ambiguity means reasonable annotators can disagree because the definition itself leaves room for judgment.

Write an annotation guide with concrete positive, negative, and boundary examples. Give uncertain cases an explicit route instead of forcing guesses. Measure inter-annotator agreement, the rate or statistic describing how consistently independent reviewers label the same examples, and investigate categories with low agreement.

label: urgent incident
positive: active customer outage or confirmed data loss
negative: routine maintenance request
uncertain: intermittent degradation without impact evidence -> expert review

Scenario: Two teams label resolved differently: one when a patch deploys, the other after 24 hours without recurrence. A model trained across both learns organizational inconsistency rather than technical resolution.

Tip: Version the label definition and record who or what produced each label. When policy changes, you can identify which training examples follow the old meaning.