Coordinating Across Tiers

Scenario: A release needs the database migrated before the web servers are updated, the load balancer drained during the change, and a smoke test afterwards. All three tiers are separate groups.

New words, in plain English

• Orchestration - coordinating changes across several tiers in a required order.
• Multi-play playbook - several plays in one file, each targeting a different group, executed in order.
• meta: end_play - stop the current play early for the hosts still in it.
• uri module - makes HTTP requests - the usual way to gate on a real health endpoint.

A playbook is a list of plays, and plays run in file order. That ordering is the orchestration primitive: one play per tier, in the order the tiers must change.

The pattern for a coordinated release is four plays. Migrate the database first, in its own play with run_once. Then roll the web tier with serial and readiness gates. Then reconfigure the load balancer. Then run a verification play that fails loudly if the result is not healthy.

Each play can carry its own settings - serial, max_fail_percentage, gather_facts - which is what makes this expressive. The migration play needs no batching; the web play needs serial: 1; the verification play needs no facts at all.

Sharing information between plays is where hostvars earns its keep. Facts gathered in an earlier play (or read from a fact cache) are available to later ones, so the load-balancer play can read each web server's port without gathering anything itself.

When a play must stop early for the remaining hosts - a precondition turned out to be false, for example - meta: end_play ends it cleanly rather than failing.

Analogy: A theatre production runs in acts, in order. The set change happens between acts, not during one. You cannot start act three while act two is still on stage - and Ansible's plays give you exactly that structure.

A worked example

# 1. Migrate the schema first - once, not per host
- name: Migrate the database
  hosts: dbservers
  gather_facts: false
  tasks:
    - ansible.builtin.command: "/opt/bin/migrate.sh {{ schema_version }}"
      run_once: true

# 2. Roll the web tier, one host at a time
- name: Deploy the web tier
  hosts: webservers
  serial: 1
  max_fail_percentage: 0
  roles:
    - webapp

# 3. Republish the backend pool once the tier is healthy
- name: Reconfigure the load balancer
  hosts: loadbalancers
  gather_facts: false
  tasks:
    - ansible.builtin.template:
        src: backends.conf.j2      # loops over groups['webservers']
        dest: /etc/lb/backends.conf
        validate: 'lb-check -c %s'
      notify: reload lb

# 4. Verify from outside, and fail loudly if it is not healthy
- name: Smoke-test the release
  hosts: loadbalancers
  gather_facts: false
  tasks:
    - ansible.builtin.uri:
        url: "http://{{ inventory_hostname }}/healthz"
        status_code: 200
      retries: 10
      delay: 3
      register: health
      until: health.status == 200

The verification play is the one teams most often omit, and it is the one that turns a deployment into a deployment you can trust. Without it, "the playbook succeeded" means every task reported success - not that the service works. A uri check with until and retries against a real health endpoint is a few lines and changes what a green run actually means.

For genuinely large estates, the same structure scales with serial percentages per tier and per-batch failure budgets. The shape does not change; only the batch sizes do.

Tip: End every deployment playbook with a verification play that fails if the service is not healthy. Otherwise 'green' only means the tasks ran.

Goal: Put this to work in the ansible-serial-rolling-update lab. Open /labs/ansible, pick ansible-serial-rolling-update, and fix the real broken project - Ansible really does SSH into four managed hosts and converge them.