Ansible Case Study: A Rolling Update Took Down the Entire Web Tier

Impact

A routine catalog release made all 18 web servers unavailable for eleven minutes. The playbook stopped the service on every host before deploying the new package. The package then failed its database compatibility check, leaving the entire tier stopped.

The load balancer had no healthy targets. Operators restored the previous package and started services manually in batches.

---

What the Playbook Did

The play targeted webservers without serial. A pre-task delegated one load-balancer command to lb01, but lacked run_once, so 18 hosts raced to rewrite the same pool file. The service stop task succeeded everywhere before any package validation occurred.

The health check used pgrep catalog. During staging tests, a supervisor helper matched that name even when the application failed readiness, so the play appeared healthy. An operator's first response was to rerun the same play with a broad retry, repeating the stop across the fleet.

Root Causes

  1. No batch-size or failure-budget policy existed.
  2. Preflight validation happened after destructive service changes.
  3. Delegation did not control cardinality.
  4. Health checks measured process-name presence rather than application readiness.
  5. The runbook recommended retry without inspecting partial state.

Safer Design

The replacement play uses serial: 2 and max_fail_percentage: 0. It asserts database compatibility and artifact checksums before removing any host from service. Each host is disabled from the load balancer, upgraded, started, and verified through /ready before it returns to the pool.

Shared pool initialization is delegated once. Per-host enable/disable remains delegated for every host, with a lock in the load-balancer API rather than concurrent file edits. The rescue path restores the prior package and ensures the host stays out of rotation if readiness fails.

CI tests the play against four ephemeral hosts, injects a failure into the second batch, and proves later batches never start. The runbook now begins with inventory, recap, and service-state evidence before any retry.

Lesson

serial is not a performance option. It defines the maximum immediate blast radius. Rolling safety also requires preflight checks, meaningful readiness, correct delegated-task cardinality, and a tested failure path.

Practice these controls in Keep the Fleet Online with serial, Delegate Once, Not Everywhere, and Recover with block, rescue, always.