Experiments and Runs

Replace mystery notebooks with evidence

By this point in the track you've already trained models, built retrieval pipelines, and tried different chunk sizes and prompts - and if you've been keeping track of all those attempts in scattered notebook cells or half-remembered notes, you've probably already felt the pain this module solves. MLflow is a tool purpose-built for recording and comparing that kind of iterative ML work in one consistent place. An MLflow experiment is simply a named group of related attempts at the same underlying task. Each individual attempt within that group is a run, and a run records four kinds of things together: parameters (the input settings you chose, like model name or chunk size), metrics (measured numbers, like retrieval recall or response time), tags (short searchable labels), and artifacts (output files, like a saved report).

import mlflow

with mlflow.start_run(run_name="chunk-size-sweep"):
    mlflow.log_param("chunk_size", 300)
    mlflow.log_param("model", "qwen2.5:0.5b")
    mlflow.log_metric("retrieval_recall", 0.82)
    mlflow.log_artifact("eval_report.json")

The rule of thumb worth internalizing is: log every input that could have changed the result, not just the final headline score. A recall number sitting by itself, with no record of which dataset version, which source-code revision, or which model tag produced it, is nearly useless a few weeks later when someone asks "wait, why did this change?"

Scenario: Two developers on the same team each report a different RAG accuracy number for what they think is "the same" pipeline, and neither can explain the gap. Pulling up their MLflow runs side by side reveals the actual difference immediately: one used a chunk size of 300 against dataset version 2, while the other used a chunk size of 800 against a stale, older copy of the dataset. Without that recorded evidence, the disagreement could have gone unresolved for a long time.