Regex Redaction in a Stream

Match structure, replace meaningfully

A regular expression (regex) is a compact, precise way of describing a pattern of text to search for, rather than one exact literal string. Getting these patterns genuinely right takes more care than it first looks: a solid email-matching pattern has to account for dots inside the local part, plus-sign aliases (like name+tag@example.com), subdomains, and a range of valid domain suffixes. A solid phone-matching pattern has to handle optional country codes, parentheses, spaces, dots, and hyphens used as separators in different regional formats. API keys are generally safest to match by their known provider-specific prefix (many providers issue keys starting with a recognizable string) combined with a conservative rule about allowed characters and minimum length.

import re
PATTERNS = {
    "EMAIL": re.compile(r"[\w.+-]+@[\w-]+\.[\w.-]+"),
    "API_KEY": re.compile(r"\bsk-[A-Za-z0-9]{20,}\b"),
}
def redact(text):
    for label, pattern in PATTERNS.items():
        text = pattern.sub(f"[REDACTED_{label}]", text)
    return text

>>> redact("contact me at ana@example.com, key sk-abc123def456ghi789jkl")
'contact me at [REDACTED_EMAIL], key [REDACTED_API_KEY]'

Replace every match with a typed marker like [REDACTED_EMAIL] or [REDACTED_API_KEY], rather than a generic [REDACTED] for everything - a typed marker preserves useful operational meaning (you can still tell what category of sensitive data was present and how often) without ever preserving the actual sensitive value itself. In code, compile the regex patterns once up front rather than recompiling them on every call, process each line or chunk of a stream as it arrives, and write only the transformed, already-redacted text onward - never the original.

Warning: Streaming introduces a genuinely tricky boundary problem that a simple line-by-line redactor can miss: an API key or email address can easily be split across two separate network chunks, arriving in two different read() calls, with the sensitive value never appearing whole in either individual chunk your regex sees. A production-grade filter needs a small rolling buffer, or a token-aware streaming layer, so a value split at an unlucky boundary doesn't slip through by having half of it already flushed out before the rest even arrives.

Regex-based redaction is fast, runs entirely locally with no network dependency, is fully auditable (you can read exactly what pattern matched what), and is completely deterministic - but it genuinely does not understand context the way a human reader does. A sequence of digits might be a harmless internal order number, or it might be something sensitive - regex alone can't reliably tell the two apart. Tune your specific patterns against the real data formats you actually see in your application, and resist the temptation to mask broad, generic digit ranges without concrete evidence they're actually sensitive.

Tip: Use re.sub with a callback function, rather than a fixed replacement string, whenever the correct replacement text depends on exactly which pattern matched. And when a redaction attempt itself fails to parse correctly, keep the original sensitive value out of the resulting diagnostic error message too - a bug report that leaks the exact secret it was trying to protect defeats the whole purpose.