A Layered Troubleshooting Method

Scenario: terraform apply fails, and the team immediately deletes .terraform, state, and the lock file—destroying evidence without finding the layer at fault.

Classify failures from outside in: CLI/configuration syntax, initialization/dependencies, backend/state/lock, provider authentication, provider API/network, resource preconditions/quotas, or post-create consistency. Reproduce with the smallest safe command, read diagnostics fully, compare versions, inspect state read-only, and consult provider/API logs. Change one variable at a time and preserve artifacts securely.

Analogy: A doctor takes history and vital signs before surgery. Deleting caches and forcing locks are procedures, not diagnoses.

A worked configuration

terraform version
terraform providers
terraform validate
terraform state list
TF_LOG=DEBUG TF_LOG_PATH=terraform-debug.log terraform plan

Write incident notes that separate symptoms, observations, hypotheses, actions, and outcomes. Terraform errors often wrap provider errors; the deepest message usually names the rejected field or permission.

Warning: Debug logs can contain credentials and full API payloads. Enable them briefly, restrict permissions, sanitize sharing, then delete according to incident policy.

Goal: Reinforce this lesson in the tf-state-drift-recovery Terraform lab. Open /labs/terraform and choose slug tf-state-drift-recovery; the lab runs real Terraform against the offline FakeCloud provider.