State and Apply Recovery Playbooks

Scenario: A terminal session disconnects midway through an apply, and nobody on the team is confident whether the remote resources it was creating actually finished, partially finished, or never started at all.

The single most important instruction for this exact moment is a restraint one: do not immediately rerun the apply, and do not immediately hand-edit state, however tempting either feels under pressure - both can make a genuinely recoverable situation much worse. Instead, work through it calmly: determine whether the original process or its remote run (if using HCP Terraform or similar) is actually still active somewhere, inspect the backend's lock status and any run history to understand what state the system believes it's in, and independently query the provider's own API or console to see what objects genuinely exist right now, separate from what Terraform's records say.

Here's the reassuring technical fact underlying this whole playbook: Terraform executes the operations in its graph incrementally, and it records each individual resource's success into state as it completes, not just at the very end of a run - so a partial failure is very often cleanly recoverable, precisely because state's identity mapping (Module 5) for whatever did finish successfully typically remains accurate even if the run itself was cut short. Once you've resolved any lingering lock or concurrent-run concern, run a fresh plan and let it tell you, based on real observed reality, what (if anything) actually needs fixing - only reach for import or targeted state operations once you have concrete evidence of a specific identity mapping problem, never as a first guess.

Analogy: After a sudden power cut interrupts a warehouse relocation mid-move, the sensible next step is inventorying both the old and new buildings to see exactly what's where before sending the moving crew back out - not immediately dispatching them again on the old plan and hoping for the best, and not throwing out the inventory records and starting from a blank sheet either.

A worked configuration

terraform state list
terraform state show fakecloud_server.api
terraform plan
# use import/state operations only for a proven mapping problem

Maintain, well before you ever need them under pressure: a working backend backup and restore procedure, a documented stale-lock resolution process (Module 5), guidance for handling provider rate limits during a recovery, and clear escalation contacts. Genuine recovery speed comes from calm, evidence-based observation and rehearsed playbooks - not from reflexively reaching for -lock=false or a force flag the moment something looks stuck.

Common mistakes in production

• Immediately rerunning an interrupted apply without first checking whether the original run or its lock is still active - this risks two concurrent writers exactly as Module 5 warned against.
• Hand-editing state to "fix" an interrupted apply's uncertainty before gathering real evidence of what state the remote objects are actually in.

Goal: Treat the plan as a change contract: understand every create, update, replacement, and destroy before approving it. Reinforce this lesson in the tf-remote-state-locking Terraform lab. Open /labs/terraform and choose slug tf-remote-state-locking; the lab runs real Terraform against the offline FakeCloud provider.