Production Ansible: Rolling Updates, Dynamic Inventory, and CI Guardrails

The First Production Control Is Batch Size

A correct playbook can still cause an outage if it restarts every server at once. serial limits each batch:

- name: Roll out the catalog service
  hosts: webservers
  serial: 1
  max_fail_percentage: 0
  pre_tasks:
    - name: Remove this host from the load balancer
      ansible.builtin.command: /usr/local/bin/lb-disable {{ inventory_hostname }}
      delegate_to: lb01
  roles:
    - catalog
  post_tasks:
    - name: Wait for readiness
      ansible.builtin.uri:
        url: http://127.0.0.1:8080/ready
        status_code: 200
      register: ready
      retries: 12
      delay: 5
      until: ready.status == 200
    - name: Return this host to service
      ansible.builtin.command: /usr/local/bin/lb-enable {{ inventory_hostname }}
      delegate_to: lb01

The health check must prove application readiness, not merely that a process exists. Decide how the load balancer recovers if the play fails between disable and enable.

---

Delegation and run_once Are Different

delegate_to changes where a task executes. run_once limits how many selected hosts initiate it. A database migration may need both, plus an explicit condition tied to the first host in the current intended run. Test behavior with serial batches because run_once semantics can surprise operators when plays are batched.

Refuse Unsafe Runs Early

Use assertions for required variables, approved environments, minimum healthy capacity, and non-empty host selection. Preview inventory and limit patterns in CI. Tags are a user interface, not a dependency solver: running --tags config can skip a prerequisite task unless tagging is designed and tested deliberately.

Dynamic inventory should group hosts from stable labels and expose the generated graph as evidence. Treat missing credentials, stale caches, and zero-host results as failures.

CI Should Verify Meaning

A strong pipeline runs syntax checks, linting, role tests, check mode where supported, a real converge on ephemeral hosts, a second idempotency run, and assertions against final service behavior. Pin collections and execution-environment dependencies. Preserve diffs and recaps without leaking vault data.

Performance comes after safety. Tune forks, fact gathering, caching, and pipelining based on measurements. Disable facts only when no task or role needs them, and never trade away batch health checks merely to shorten a rollout.

Practice the production controls in Keep the Fleet Online with serial, Delegate Once, Not Everywhere, Refuse an Unsafe Run, and Repair Dynamic Inventory.