Choosing How Much Failure Is Acceptable

Scenario: A schema migration was rolled out across a fleet. Three hosts failed. The other forty-seven upgraded successfully, and the application now had two incompatible schema versions in production.

New words, in plain English

• any_errors_fatal - abort the entire play the moment any host fails.
• max_fail_percentage - abort once more than this share of hosts in a batch have failed.
• serial - process hosts in batches rather than all at once.
• Partial rollout - a state where some hosts have the change and others do not.

Ansible's default - one host failing does not stop the others - is right for some changes and catastrophic for others. The decision is about whether a partial rollout is acceptable, and it should be made deliberately per playbook rather than inherited by accident.

For an independent, additive change - installing a monitoring agent across 500 hosts - a partial rollout is fine. Let the failures happen, report them, fix them separately.

For a coordinated change where hosts must agree - a schema migration, a protocol version bump, a shared cache format - a partial rollout is worse than no rollout at all. That is the scenario above: forty-seven successes created the outage, not the three failures.

any_errors_fatal: true stops everything at the first failure. max_fail_percentage: 25 allows a proportion before aborting, and is evaluated per batch when combined with serial - so serial: 10 with max_fail_percentage: 0 stops the rollout at the end of the first batch containing any failure, which is precisely the canary behaviour you want.

And remember force_handlers: true: without it, a play that aborts leaves notified handlers unrun, so a configuration change can land without its restart.

Analogy: A vaccination programme can proceed if a few people miss their appointment. Switching a railway to a new signalling standard cannot - half the network on the old standard and half on the new is far more dangerous than not starting.

A worked example

# Independent, additive change - partial rollout is fine
- name: Install the monitoring agent
  hosts: all
  tasks:
    - ansible.builtin.package:
        name: node-exporter
  # defaults: failures are reported, other hosts continue

# Coordinated change - a partial rollout is the outage
- name: Apply the schema migration
  hosts: dbservers
  serial: 1
  max_fail_percentage: 0        # ANY failure stops the rollout
  force_handlers: true          # but still flush pending handlers
  tasks:
    - name: Run the migration
      ansible.builtin.command: /opt/bin/migrate.sh {{ schema_version }}

# Canary: first one host, then the rest in batches
- name: Roll out the release
  hosts: webservers
  serial: [1, 5, '25%']
  max_fail_percentage: 0

serial accepts a list, which produces a genuine canary pattern: serial: [1, 5, '25%'] does one host, then five, then quarter-batches. Combined with max_fail_percentage: 0 the rollout stops at the end of whichever batch first contained a failure - so a bad release reaches one host, not the estate.

The honest question to ask of any playbook is: if this fails on host seventeen of fifty, what state is the system in? If the answer is "fine, we fix seventeen", the defaults are correct. If the answer is "we now have two incompatible versions in production", you need any_errors_fatal or a zero failure budget, and you need it before the rollout rather than after.

Tip: Ask of every playbook: if this fails half-way, is the resulting state acceptable? The answer chooses your failure strategy for you.

Goal: Put this to work in the ansible-block-rescue-always lab. Open /labs/ansible, pick ansible-block-rescue-always, and fix the real broken project - Ansible really does SSH into four managed hosts and converge them.