What Must Be Masked?

Classify data before logging it

Personally identifiable information (PII) is any data that identifies, or that can reasonably be linked back to, a specific individual person - common examples include email addresses and telephone numbers, but the category is broader than just those two. Credentials, like API keys, aren't always formally classified as PII under privacy law, but they deserve at least the same level of protective handling, because possessing one grants real access to a real system - leaking one isn't just an embarrassing privacy slip, it's a live security incident.

The reason this module matters specifically for LLM applications is that a model can repeat sensitive text back to you from anywhere it appeared in a prompt: from the user's own typed input, from a retrieved document (recall the RAG modules), from a tool's response, or from earlier turns of conversation history it's still holding onto. That means redaction genuinely needs to sit on every route data can travel toward storage or display - application logs, the tracing system from earlier in this phase, an experiment tracker like MLflow, saved chat transcripts, and text streamed live to a browser - not just the one obvious place you thought to check first.

Analogy: A mailroom blacks out account numbers on a form before photocopying it, never after. It has to redact before the copier runs, because once copies already exist and have started circulating, reliably tracking down and destroying every single one of them afterward is essentially impossible.

Scenario: A model's response happens to include an email alias, an international phone number in an unusual format, and two separate provider API keys. The application's logging code only has simple pattern rules for a plain name@example.com format and a plain 555-555-5555 format - so all four of those sensitive values sail straight through untouched, and the resulting log file itself becomes a genuine data-breach artifact sitting on disk.

When building and testing any of this, always use synthetic canary values made up specifically for the test - never real customer data and never a live, working credential. And if a real secret ever does actually reach a log despite your defenses, understand clearly that redaction alone is not a sufficient response at that point: the correct next step is to revoke or rotate that specific credential immediately and follow your organization's incident process, since the secret must be treated as compromised the moment it touched an insecure location.