Security Failures and Recovery

Scenario: A secret is accidentally printed to a CI log during a run, and someone's first instinct is to delete the log and consider the incident closed - without remembering that the same value likely also exists in Terraform state and possibly a saved plan file.

A real secret-exposure incident needs a specific, ordered response, and getting the order wrong genuinely matters. First, revoke or rotate the exposed credential itself - this is the single action that actually neutralizes the exposure; everything else is cleanup around it, and deleting a log without doing this first (as in the scenario above) leaves the real credential still valid and usable by whoever saw it. Next, contain access to whatever system the leak occurred in, and preserve appropriate audit evidence before you start cleaning anything up - you'll want to know exactly how and when the exposure happened. Then inventory every copy: given everything you learned in the first section of this module, that almost certainly means checking state (including older state versions if your backend keeps history), any saved plans, debug logs, caches, and backups - not just the one log file someone happened to notice.

Finally, fix the actual cause: remove the value from wherever it entered configuration, and redesign the injection path so the same class of leak can't recur (often: move toward the short-lived, reference-based credentials from earlier in this module). Be honest that directly editing state to strip out a leaked secret is genuinely risky surgery - use Terraform's own supported state-replacement or migration procedures, and involve whoever administers your backend, rather than hand-editing a state file to "fix" it.

Analogy: If a photograph of your house key ends up posted publicly, deleting the photo from the internet does not secure your house - the person who already saw it can still have a copy cut. The actual fix is changing the lock (revoking/rotating the credential) first; only after that does it make sense to go track down and clean up every copy of the photo, and fix whatever process let the photo get taken in the first place.

A worked configuration

# Incident sequence (conceptual):
# 1 revoke/rotate  2 contain  3 inventory copies
# 4 redesign flow   5 sanitize/expire artifacts  6 verify

Practice this response before you ever need it for real: know in advance whether your backend keeps versioned state history, where its access logs live, how long your CI system retains artifacts, who actually owns rotation for each class of credential you use, and who to escalate to. In a genuine incident, speed of rotation and containment matters enormously more than any illusion that secrets can simply never leak - plan for the leak, not just for prevention.

Common mistakes in production

• Deleting a log that leaked a secret and considering the incident resolved, without first revoking or rotating the actual exposed credential - the credential itself is still valid and usable.
• Hand-editing a state file to remove a leaked value instead of using Terraform's supported state-replacement procedures - this risks corrupting the state's integrity on top of the original incident.

Goal: Do not paste raw Terraform debug logs, state fragments, or plan JSON into a public issue tracker or shared channel while investigating an incident, even with good intentions - sanitize locally first, and share only through an approved, access-controlled sensitive channel. Reinforce this lesson in the tf-sensitive-outputs Terraform lab. Open /labs/terraform and choose slug tf-sensitive-outputs; the lab runs real Terraform against the offline FakeCloud provider.