Experiments and Multiple Comparisons

Choose the test before seeing the winner

A randomized A/B test assigns comparable units to a control or candidate system, helping balance both known and unknown confounders. Define the unit, primary metric, minimum meaningful effect, stopping rule, and analysis plan before looking at results. Repeatedly peeking and stopping when a number looks favorable inflates false discoveries.

Testing many metrics, subgroups, models, and thresholds also raises the chance that something looks significant by luck. This is the multiple comparisons problem. Corrections, held-out confirmation, and clearly separating exploratory from confirmatory analysis help keep accidental winners from becoming claims.

20 independent tests at a 5% false-positive threshold
chance of at least one false alarm = 1 - 0.95^20 ≈ 64%

Analogy: Roll enough dice and eventually a streak looks miraculous. It becomes evidence only if you declared the exact streak and stopping rule before rolling.

Note: Statistical tools do not rescue a broken experiment. Interference between users, logging changes, sample-ratio mismatches, and novelty effects all require operational investigation.