Profile Before You Clean

Inspection comes before repair

Real data inherits every mistake and convention of the systems that produced it: time zones disagree, numbers arrive as strings, old services use different category names, retries create duplicate events, and sentinel values such as -999 secretly mean missing. Data profiling is the systematic first look at shape, types, missingness, ranges, category counts, and duplicates before any transformation changes the evidence.

import pandas as pd
df = pd.read_csv('events.csv')
print(df.shape)
print(df.dtypes)
print(df.isna().sum())
print(df.nunique())
print(df.describe(include='all'))
print(df.duplicated().sum())

The declared type and the semantic meaning are different. A numeric region_code may be a category, while an amount column loaded as text may need numeric conversion. A data dictionary records what each field means, its unit, valid range, owner, and source. Without that context, 0 could mean zero, false, unknown, or not applicable.

Analogy: Profiling is the building survey performed before renovation. Painting immediately may hide cracks, but it cannot tell you whether the foundation moved.

Tip: Save the profiling report. Comparing today's missing rates and category counts with a known-good report often reveals upstream changes before model metrics fail.