Cleaning Without Hiding Problems

Make every repair explicit

Once you know a numeric-looking column is secretly text, the fix is to convert it deliberately and let Pandas tell you exactly which values it couldn't understand, rather than guessing silently:

df["income"] = pd.to_numeric(df["income"], errors="coerce")
df["income"].isna().sum()

347

errors="coerce" means: convert everything that looks like a real number, and turn anything it can't parse (like "unknown" or a stray typo) into NaN instead of crashing. Notice the missing count went up, from 312 to 347 - those extra 35 rows had text pandas couldn't read as a number. This is safer than a column that quietly mixes numbers and strings, because from here on every tool downstream can rely on income actually being numeric.

Now you have missing values and must decide what to do with them - a step called imputation: replacing a missing value with a reasonable estimate instead of leaving a hole. A common, simple choice is the median (the middle value once everything is sorted), because a handful of extreme outliers (a billionaire's income row) can drag the mean (the plain average) far from what's typical.

median_income = df["income"].median()
df["income"] = df["income"].fillna(median_income)

Warning: There's a subtle but important trap here: that median must be calculated only from the training portion of your data (covered in the next module), never from rows you're holding back to test the model. Learning the median from data the model will later be "tested" on is a small form of information leaking across a boundary that's supposed to be sealed - it makes your evaluation quietly optimistic.

Models expect numbers, but a lot of real-world features are categories, like a subscription plan of basic, pro, or enterprise. One-hot encoding turns one category column into several 0/1 columns, one per possible value:

pd.get_dummies(df["plan"], prefix="plan").head(3)

   plan_basic  plan_enterprise  plan_pro
0           1                0         0
1           0                0         1
2           0                1         0

Do not instead encode basic=0, pro=1, enterprise=2 unless the categories genuinely have that kind of order (small/medium/large would be fine) - otherwise you're quietly telling the model that enterprise is "twice as much" as pro, a relationship that doesn't exist and that the model will treat as real.

Feature engineering is the broader craft of building useful inputs from raw fields - for example, computing account_age_days from a signup date instead of feeding the model a raw date string it can't use. Any feature you invent must be something the application can actually compute at the moment it needs to make a prediction, not something only available afterward (more on that danger in the next section), and it must be computed the exact same way during training and in production.

Warning: fillna(0) is not universal cleaning, and it's one of the most common beginner mistakes. Zero may be a real, meaningful value - a real age of zero doesn't make sense, but a real price or count of zero often does. Blindly filling every gap with 0 erases the difference between "unknown" and "genuinely zero," and the model will happily learn a false pattern from it.