Samples Naturally Vary

A different handful gives a different answer

A random sample gives each eligible population member a defined chance of selection. Even with fair sampling, two samples produce different means and metrics by chance. Sampling variability is this ordinary sample-to-sample movement; it shrinks as sample size grows, though larger biased samples remain biased.

import numpy as np
population = np.array([0, 0, 0, 1, 1, 1, 1, 1])
for seed in range(3):
    print(np.random.default_rng(seed).choice(population, 4).mean())
# Different samples can report different positive rates.

A sampling distribution is the distribution of a statistic across many hypothetical repeated samples. It lets us reason about how uncertain a sample mean, accuracy, or difference is. We rarely collect infinite repetitions, so resampling methods and statistical theory approximate that uncertainty.

Scenario: A new model beats the baseline on 20 test cases by one correct answer. That apparent five-point improvement may reverse with another small sample; treating it as a proven win ignores sampling variability.

Warning: A random seed reproduces one sample. It does not prove that this particular sample represents all plausible samples.