Terraform Drift Case Study: The Console Change That Broke a Production Plan

Monday, 09:17 - A Routine Plan Is Not Routine

Atlas Analytics opens a pull request to add one DNS record for its reporting API. The Terraform plan unexpectedly proposes shrinking the production database:

  # fakecloud_database.warehouse will be updated in-place
  ~ size_gb = 800 -> 400

  # fakecloud_dns_record.reporting will be created
  + resource "fakecloud_dns_record" "reporting" { ... }

Plan: 1 to add, 1 to change, 0 to destroy.

The author changed only dns.tf. The database diff is not noise; it is evidence that configuration and the provider disagree.

---

09:21 - Read the Three Sources of Truth

Engineer Luis checks configuration:

rg -n 'size_gb|warehouse' .
terraform state show fakecloud_database.warehouse

HCL declares 400 GB. After the plan's refresh, state records 800 GB. The FakeCloud inventory also reports 800 GB. Provider audit history reveals a console resize Friday at 22:48 by the data on-call engineer.

That resize was intentional. A late batch had exhausted storage, and expanding the database restored ingestion. The incident ticket was resolved, but nobody reconciled Terraform afterward.

The normal plan refreshed state in memory, compared refreshed reality with configuration, and proposed making reality match HCL. Terraform is behaving correctly.

---

09:28 - Reject the Tempting Wrong Fixes

The first suggestion is:

terraform apply -refresh-only

But state is already discovering 800 GB during normal planning. A refresh-only apply would persist that observation; it would not change configuration from 400 to 800. The next normal plan would still propose 800 to 400.

The second suggestion is:

lifecycle {
  ignore_changes = [size_gb]
}

That would stop Terraform managing database capacity indefinitely. No external autoscaler owns this field; the manual change was an emergency exception. Ignoring it would conceal future accidental resizing.

The third suggestion is terraform plan -refresh=false. That would temporarily hide provider-side drift and produce an incomplete plan. It is useful in limited diagnostic or performance contexts, not as drift remediation.

---

09:34 - Decide the Desired State

The team asks the question Terraform cannot answer: should production be 400 or 800 GB?

Database metrics show 612 GB used and a forecast of 690 GB before the next retention cycle. Shrinking is unsafe and may not even be supported by the provider. The live 800 GB setting is the new desired state.

Luis updates configuration:

resource "fakecloud_database" "warehouse" {
  name    = "analytics-warehouse-prod"
  engine  = "postgres"
  size_gb = 800

  lifecycle {
    prevent_destroy = true
  }
}

The pull request links the Friday incident, capacity graph, cost change, and database owner's approval. Emergency reality is being converted into reviewed code rather than merely accepted into state.

---

09:43 - Produce a Clean Plan

terraform fmt -check
terraform validate
terraform plan -out=reconciled.tfplan
terraform show reconciled.tfplan

Now the output is scoped to the requested DNS change:

Plan: 1 to add, 0 to change, 0 to destroy.

After approval, CI applies that exact plan. The database remains at 800 GB and the DNS record is created.

---

Why Drift Detection Had Been Silent

Atlas ran Terraform only when infrastructure pull requests were opened. The production root had gone 19 days without a plan, so Friday's manual change remained undiscovered until Monday's unrelated change.

The team adds a nightly read-only drift job:

terraform init -input=false
terraform plan -detailed-exitcode -input=false -no-color > plan.txt
case $? in
  0) echo "No drift or configuration change" ;;
  2) echo "Changes detected"; exit 2 ;;
  *) echo "Plan failed"; exit 1 ;;
esac

Exit code 2 means the plan contains changes; exit code 1 means Terraform failed. The job opens an alert but never auto-applies. A human still decides whether to revert reality to configuration or update configuration to reflect an approved emergency change.

Post-Mortem

Root cause: an emergency console resize was not followed by an IaC reconciliation change. Contributing factors: no scheduled drift detection and an incident template with no infrastructure-as-code follow-up field.

The new process allows emergency console access through a time-limited role, records audit events, requires a follow-up owner, and alerts on nightly plan differences. The objective is not to ban urgent manual action; it is to ensure manual action does not become invisible configuration debt.

Practice deciding between restoring declared configuration and accepting intentional drift in State Drift Recovery.