Confidence Intervals Show Precision

An estimate needs an uncertainty range

A sample metric is an estimate, not a permanent property. A confidence interval gives a range generated by a procedure that would contain the true population quantity at a stated rate across repeated samples. A 95% confidence interval does not literally mean a fixed true value has a 95% chance of sitting in this already-computed interval; it describes the long-run coverage of the method.

candidate accuracy: 82%
95% interval: 78% to 86%
baseline accuracy:  80%
95% interval: 77% to 83%

Wide intervals signal limited evidence, often from few or highly variable observations. Overlapping intervals do not by themselves settle every comparison, but the example clearly warns against proclaiming a decisive two-point win. For ML metrics with awkward formulas, bootstrap resampling repeatedly samples test rows with replacement and recomputes the metric to approximate its uncertainty.

Analogy: A weather estimate of 20°C is more useful when accompanied by likely 19-21°C than by somewhere from 5-35°C. Both have the same center; one is much more precise.

Tip: Report the point estimate, interval, sample size, and sampling unit together. Resampling rows is misleading when rows from the same user or machine are dependent; resample whole groups instead.