serial: Never Take the Whole Tier Down

Scenario: The 09:40 deploy took the entire web tier offline for forty seconds. Both web servers drained at the same time, because the play processed all hosts in parallel.

New words, in plain English

• serial - how many hosts to process in each batch. Without it, Ansible works on all hosts at once (up to forks).
• Batch - one group of hosts processed together through the whole play before the next batch starts.
• Drain - taking a host out of service so traffic stops reaching it.
• Canary - a very small first batch, so a bad change reaches one host rather than all of them.

By default a play runs task-by-task across every host simultaneously - task one on all hosts, then task two on all hosts. For a rolling update that is exactly wrong: the "drain" task drains every host at once, and the tier goes dark until the "restore" task runs.

serial changes the unit of work from the task to the batch. serial: 1 runs the entire play against one host, then the next. Each host is drained, updated and restored before the following one is touched, so the tier never loses more than one member.

serial accepts a list for a canary pattern: serial: [1, 5, '25%'] does one host, then five, then quarter-batches. Combined with max_fail_percentage: 0, a bad release stops after the first host.

The cost is time - serial: 1 across fifty hosts is fifty sequential passes. The trade is availability against duration, and the right answer depends on how much capacity you can lose. serial: '25%' on a fleet that can survive losing a quarter is often the sensible middle.

One subtlety worth knowing: serial batches also bound max_fail_percentage, which is evaluated per batch, not across the whole play.

Analogy: Repainting a bridge by closing every lane at once technically finishes sooner. Closing one lane at a time takes longer and keeps the bridge open, which is the entire point of the bridge.

A worked example

- name: Roll out release {{ new_version }}
  hosts: webservers
  serial: 1                    # one host at a time - tier stays up
  max_fail_percentage: 0       # any failure aborts the rollout
  vars:
    health_file: /srv/app/health

  tasks:
    - name: Take this node out of service
      ansible.builtin.copy:
        dest: "{{ health_file }}"
        content: "draining\n"

    - name: Wait for in-flight requests to finish
      ansible.builtin.wait_for:
        timeout: 5

    - name: Install the new release
      ansible.builtin.unarchive:
        src: "app-{{ new_version }}.tar.gz"
        dest: /srv/app

    - name: Wait for the service to answer again
      ansible.builtin.wait_for:
        port: 8080
        state: started
        timeout: 60

    - name: Put this node back in service
      ansible.builtin.copy:
        dest: "{{ health_file }}"
        content: "ok\n"

wait_for is what makes a rolling update honest. Without a readiness gate, the play marks a host restored the instant the file is written - not when the service is actually able to serve. The next batch then starts against a tier with one member still warming up, and with serial: 1 on a two-node tier that can mean effectively zero capacity.

Gate on something real: wait_for on the listening port, or uri polling a health endpoint with retries and delay. "The task finished" and "the service is ready" are different facts, and only one of them matters to your users.

Warning: Without a readiness gate, a rolling update can still cause an outage: the play moves to the next host before the previous one is genuinely serving again.

Goal: Put this to work in the ansible-serial-rolling-update lab. Open /labs/ansible, pick ansible-serial-rolling-update, and fix the real broken project - Ansible really does SSH into four managed hosts and converge them.