Terraform Stale State Lock Incident: Unblocking a Production Deploy Safely

14:06 - The Deploy Stops

Northstar's checkout API is dropping requests after a certificate rotation. The fix is already approved: update the certificate binding on two edge instances. The production Terraform job starts, waits five minutes, then fails:

Error: Error acquiring the state lock

Lock Info:
  ID:        8f6d3d91-58cc-4de8-a38b-d145ad92c220
  Path:      states/prod/edge.tfstate
  Operation: OperationTypeApply
  Who:       ci-runner@gha-runner-17
  Version:   1.9.8
  Created:   2026-07-11 13:42:18.392 +0000 UTC
  Info:      pipeline 8841

The fastest command is terraform force-unlock. It is not the first command. A live apply and a second apply writing the same state can lose bindings or act on stale assumptions.

---

14:08 - Establish Ownership

On-call engineer Maya checks the CI system. Pipeline 8841 was canceled when its runner lost network connectivity at 13:44. The job UI says canceled, but that alone is insufficient: a child Terraform process could still be running.

The team checks the runner controller and process list. Runner 17 was terminated and replaced. Backend audit logs show the last state write at 13:43 and no lease renewal or API call from that identity afterward. The deployment channel has no manual runs in progress.

Maya records four facts in the incident thread:

Exact backend path: states/prod/edge.tfstate.
Lock ID and creation time.
Original runner no longer exists.
No current Terraform process or concurrent maintenance owns the lock.

This is now evidence of a stale lock, not merely an inconvenient lock.

---

14:13 - Reproduce With a Timeout

From a clean incident checkout pinned to the approved commit:

terraform init -reconfigure   -backend-config=prod.backend.hcl
terraform workspace show
terraform plan -lock-timeout=60s -out=incident.tfplan

The plan cannot acquire the same lock. The timeout was useful when a legitimate apply might finish shortly; it cannot repair an orphaned lock.

They do not use -lock=false. That flag would run without mutual exclusion and turn a recoverable metadata problem into a concurrency risk.

---

14:16 - Back Up What Can Be Read

The backend allows state reads while the stale write lock exists:

umask 077
terraform state pull > prod-edge-pre-unlock.json
terraform state list | sort > prod-edge-addresses.txt

The snapshot serial is 184 and contains 37 resources, including both edge instances and the certificate. The encrypted incident workspace retains the backup under restricted access.

Maya compares backend object history. Serial 184 matches the latest stored version; there is no failed local-state artifact attached to pipeline 8841. If the original apply had changed infrastructure but failed before persisting state, unlocking would be only the beginning: the next refreshed plan would reveal that partial change.

---

14:20 - Unlock the Exact State

With a second platform engineer confirming the evidence, Maya runs:

terraform force-unlock 8f6d3d91-58cc-4de8-a38b-d145ad92c220

Terraform asks for confirmation and reports that the lock was released. force-unlock removes lock metadata; it does not modify remote infrastructure.

She deliberately omits -force so the interactive prompt provides one last check. In automation, -force can be necessary, but the runbook requires the same ownership proof and peer approval.

---

14:22 - Assume the Interrupted Apply Was Partial

The team creates a fresh plan rather than applying the earlier failed job's intent blindly:

terraform plan -out=post-unlock.tfplan
terraform show post-unlock.tfplan

Output summary:

Plan: 0 to add, 2 to change, 0 to destroy.

  # fakecloud_instance.edge["a"] will be updated in-place
  ~ certificate_id = "cert-old" -> "cert-2026-07"

  # fakecloud_instance.edge["b"] will be updated in-place
  ~ certificate_id = "cert-old" -> "cert-2026-07"

The refresh shows pipeline 8841 changed neither instance before disconnection. The engineer verifies there are no replacements, applies the saved plan, and confirms both endpoints serve the renewed chain. Error rate returns to baseline at 14:29.

---

Root Cause

The backend used a lock record whose cleanup depended on the Terraform client completing its shutdown path. The runner termination interrupted that cleanup. The pipeline cancellation handler stopped the job but did not verify lock release.

The team made three changes:

Graceful runner termination now gets a 90-second window before forced deletion.
Apply jobs use a bounded -lock-timeout and expose lock metadata in the alert.
The stale-lock runbook requires runner/process proof, backend path confirmation, state backup, peer approval, and a new refreshed plan.

They rejected an automatic job that force-unlocked every lock older than 15 minutes. Long applies are legitimate, and lock age does not prove abandonment.

Lessons

A lock is a safety mechanism, not an error to bypass. First identify its owner. Use -lock-timeout for contention, force-unlock LOCK_ID only for a proven orphan, and never substitute -lock=false during an incident.

You can rehearse the exact failure in Remote State Locking, which includes a local HTTP backend, a stale lock, and a safe recovery path.