local-exec, remote-exec, and file
Scenario: A local-exec provisioner command works perfectly on one engineer's laptop, then fails mysteriously in the CI pipeline, because the tool it invokes, the default shell, and the file paths available are all subtly different between the two environments.
Provisioners come in a few flavors, and each one's execution environment - where the command literally runs - is worth being precise about, because that's exactly what tripped up the scenario above. local-exec runs the given command on whatever machine is actually running Terraform itself (your laptop, or a CI runner) - not on the resource being created at all. remote-exec instead opens a connection (commonly SSH) to the newly-created resource and runs commands there. file transfers a local file's content up to the remote resource. Because local-exec's "local" machine changes depending on who or what is running Terraform, any script it invokes needs to work identically across every environment it might ever run in - explicit interpreter, no assumptions about tools being pre-installed, no hardcoded personal paths.
Creation-time provisioners (the default) and destroy-time provisioners (when = destroy) behave differently in a way worth knowing: a destroy-time provisioner runs before the resource is actually destroyed and, critically, only has access to the resource's own values (via self) - not to other resources' current values, since those may already be gone by the time destruction is being planned. on_failure = continue lets Terraform proceed past a failed provisioner rather than halting - useful for genuinely optional steps, but it can also quietly leave infrastructure in an incomplete state while reporting overall success, so use it deliberately, not as a way to make errors disappear.
Analogy: A local-exec provisioner hires the person standing right next to the blueprint table to do the work - not someone physically inside the new building. If you change who's standing at the table (a different engineer's laptop, a different CI runner), you've changed which tools, shell, and paths that worker actually has on hand, even though the instructions you gave them look identical.
A worked configuration
provisioner "local-exec" {
command = "./notify.sh ${self.id}"
interpreter = ["/bin/bash", "-c"]
}
If you genuinely can't avoid a provisioner, minimize the damage it can do: make the command idempotent (safe to run more than once), pin its runtime environment explicitly rather than assuming ambient tools are present, never pass a secret as a literal argument in the command string (it can end up in logs or process listings), capture output somewhere auditable, set a bounded timeout so a hang doesn't block the whole apply indefinitely, and document exactly how to recover manually if it fails partway. Keep the scope small enough that you could realistically replace it with a proper provider resource or image-based approach later, rather than letting it grow into critical, hard-to-remove infrastructure logic.
Common mistakes in production
- Writing a
local-execscript that assumes a specific tool or shell is installed on "whatever machine runs Terraform" without pinning the interpreter or checking availability - works on one laptop, breaks in CI. - Passing a secret value directly as a literal argument in a provisioner's command string - it can end up captured in shell history, process listings, or logs.
Goal: Treat the plan as a change contract: understand every create, update, replacement, and destroy before approving it. Reinforce this lesson in the tf-terraform-data-triggers Terraform lab. Open/labs/terraformand choose slugtf-terraform-data-triggers; the lab runs real Terraform against the offline FakeCloud provider.