Provenance and Lineage

Know where every value came from

Data provenance describes the origin of data - source system, collection time, method, and ownership. Lineage describes the transformations that carried it from raw source to a model-ready table. These records let a team explain a suspicious feature, reproduce a dataset, and identify every model affected by an upstream correction.

billing.transactions v7
  -> remove test accounts
  -> aggregate per customer at prediction_time
  -> join support_counts using customer_id
  -> features_2026_09.parquet

A good lineage entry records code revision, input versions, row counts before and after each step, schema checks, and timestamps. The MLflow and DVC modules later in this track show tools for tying code, data, and experiment evidence together; the habit begins here with an explicit transformation story.

Scenario: A feature's average changes overnight. With lineage, the team traces it to one billing job that switched from dollars to cents. Without lineage, they may retrain a model on corrupted values and call the resulting change adaptation.

Goal: Practice these profiling and contract habits in the existing ml-data-hygiene challenge, where invalid numeric values and leakage must be diagnosed rather than hidden.