What Statistical Significance Means

Evidence against a specific null

A hypothesis test begins with a null hypothesis, usually that an observed difference could come from a no-effect process. A p-value is the probability, assuming that null and the test's assumptions are true, of observing a result at least as extreme as the one measured. A small p-value is evidence against the null; it is not the probability that the null is true.

Statistical significance means a chosen test crossed a predeclared threshold such as 0.05. It does not prove practical importance, a large effect, causality, correct data, or future reproducibility. With enormous samples, trivial differences can be significant; with tiny samples, important effects may remain uncertain.

Scenario: A latency optimization saves 0.2 milliseconds across ten million requests and produces a tiny p-value. If users cannot perceive it and infrastructure cost does not change, the result is statistically detectable but practically unimportant.

effect size: how large is the difference?
uncertainty: how precisely was it measured?
p-value: how surprising under the stated no-effect model?

Warning: p = 0.03 does not mean there is a 97% chance the new model is better. That interpretation reverses the conditional probability.