Why Values Go Missing

Absence has a cause

A missing value marks unavailable information, but missingness can arise in different ways. It may be roughly random, such as a packet lost independently of its contents. It may depend on observed facts, such as older devices omitting a field. Or it may depend on the missing value itself, such as high-income respondents being less likely to disclose income. These mechanisms affect which repair is credible.

Dropping every incomplete row can shrink the dataset and systematically remove particular groups. Imputation fills missing entries using a rule learned from available data. Median imputation is robust for skewed numeric values; a most-frequent category or explicit unknown category can serve categorical fields. A missingness indicator is an added 0/1 feature recording that the original value was absent, preserving a signal that the act of being missing may carry.

df['income_missing'] = df['income'].isna().astype(int)
df['income'] = df['income'].fillna(training_median)

Warning: Calculate training_median on training data only. Learning it from the full dataset lets information from validation or test rows influence preprocessing.

Scenario: A sensor stops reporting temperature only when it overheats. Filling every gap with the normal average and discarding the missing flag erases the strongest warning signal in the system.