Duplicates and Outliers Need Context

Unusual does not automatically mean wrong

A duplicate can be an accidental repeated record or two legitimate events with identical values. Deduplicate using a defensible business key - for example, one event ID - rather than deleting every row that happens to look alike. Decide which copy wins using timestamps or source priority, and measure how many rows the rule removes.

An outlier is a value unusually far from most others. It might be a typo, a changed unit, fraud, a rare but valid customer, or the precise incident the model must learn to detect. Statistical rules such as distance from the median can flag candidates, but domain evidence must decide whether to repair, cap, exclude, or retain them.

q1, q3 = df['latency_ms'].quantile([0.25, 0.75])
iqr = q3 - q1
suspect = df[~df['latency_ms'].between(q1 - 1.5*iqr, q3 + 1.5*iqr)]

Scenario: A latency value of 65000 looks extreme. Logs reveal it is a real dependency timeout, not a millisecond/second typo. Removing it would teach the reliability model that the failure state never occurs.

Warning: Never silently drop inconvenient rows until a chart looks clean. Record the rule, removed count, affected groups, and downstream metric change so cleaning remains auditable.