block, rescue, always: Failing Safely

Scenario: A release was corrupt on one node. The play used ignore_errors: true, went green, left current pointing at an empty directory, and the deploy lock was still held the next morning - blocking every subsequent release.

New words, in plain English

• block - a group of tasks treated as one unit.
• rescue - tasks that run only if a task in the block failed.
• always - tasks that run afterwards regardless of success or failure.
• ignore_errors - marks a failure as ignorable and continues - it does NOT clean anything up.
• Rollback - returning to the last known-good state after a failed change.

block/rescue/always is try/catch/finally for infrastructure, and it is the difference between a failed deployment and a broken estate.

block holds the work. rescue runs only if something in the block failed - this is where rollback lives. always runs either way - this is where cleanup lives, and it is the part that releases the lock.

Crucially, a handled failure means the host is not marked failed. The play continues, the recap shows rescued=1, and the host stays in the run. That is what lets one node fail its verification, roll itself back, and let the rest of the rollout proceed.

Contrast this with ignore_errors: true, which is what the scenario used. It suppresses the failure and continues to the next task. It does not roll back, it does not clean up, and it still prints the task red. It converts a loud failure into a quiet broken state - which is strictly worse than failing.

Inside rescue, ansible_failed_task and ansible_failed_result tell you which task failed and why, so the rollback can log something useful rather than a generic message.

Analogy: ignore_errors is a surgeon noticing a problem and carrying on regardless. rescue is the plan for what to do when the problem appears. always is closing the patient up either way - which is not optional.

A worked example

- name: Deploy release {{ release }}
  block:
    - name: Take the deploy lock
      ansible.builtin.copy:
        dest: "{{ app_dir }}/deploy.lock"
        content: "locked by ansible\n"

    - name: Stage and activate the new release
      ansible.builtin.file:
        src: "{{ app_dir }}/releases/{{ release }}"
        dest: "{{ app_dir }}/current"
        state: link

    - name: Verify the activated release
      ansible.builtin.command: "{{ app_dir }}/verify.sh {{ app_dir }}/current"
      changed_when: false

  rescue:
    - name: Roll back to the previous release
      ansible.builtin.file:
        src: "{{ app_dir }}/releases/{{ previous_release }}"
        dest: "{{ app_dir }}/current"
        state: link

    - name: Record why we rolled back
      ansible.builtin.lineinfile:
        path: "{{ app_dir }}/../var/log/deploy-rescue.log"
        create: true
        line: "rolled back {{ release }}: {{ ansible_failed_task.name }}"

  always:
    - name: Release the deploy lock, whatever happened
      ansible.builtin.file:
        path: "{{ app_dir }}/deploy.lock"
        state: absent

A caution about rescue: it is only useful if the rollback itself is reliable. A rescue block that assumes the previous release still exists, or that a symlink is in a particular state, can fail in its own right - and a failure inside rescue is not caught by anything. Keep rescue logic simple and defensive, and prefer restoring a recorded previous state over reconstructing one.

Blocks nest, so a large deployment can have an outer always for the lock and inner blocks for individual stages. Nest sparingly; two levels is usually the point at which a reader stops being able to hold the control flow in their head.

Warning: ignore_errors: true does not clean up, does not roll back, and still marks the task failed. If you need recovery, you need rescue. If you need cleanup, you need always.

Goal: Put this to work in the ansible-block-rescue-always lab. Open /labs/ansible, pick ansible-block-rescue-always, and fix the real broken project - Ansible really does SSH into four managed hosts and converge them.