Ansible Incident Case Study: The Handler That Never Restarted the Service

Incident Summary

At 16:10, the checkout team changed the service listener from port 8080 to 8081 and ran its normal Ansible deployment. The play recap was green and reported one changed task. At 16:24, monitoring showed the process still listening on 8080. New instances registered on 8081, creating inconsistent routing and intermittent 502 responses.

Traffic stabilized at 16:41 after an operator restarted the service manually. No orders were lost, but 6.8 percent of checkout requests failed during the incident.

---

Investigation

The rendered configuration was correct on every host. File modification times matched the deployment, and syntax validation had passed. The running process start time predated the change.

The task contained:

notify: restart checkout

The role handler had been renamed during cleanup:

- name: Restart Checkout
  ansible.builtin.service:
    name: checkout
    state: restarted

The team's wrapper suppressed warning output and only checked the process exit status. Its smoke test read the configuration file rather than connecting to the application. The automation therefore proved declared state, not effective runtime state.

Contributing Factors

• Handler notification was treated as an informal string rather than a stable interface.
• The test environment used a process that reloaded configuration automatically, masking the missing restart.
• Production verification checked a file and PID, not the serving port.
• The play had no post-handler readiness assertion.
• Review focused on YAML validity and missed behavior.

Remediation

The role now gives the handler a stable listen: restart checkout topic. A post-task flushes handlers before a readiness probe when later deployment steps depend on the new port. Verification connects to the expected endpoint and checks a version/configuration fingerprint returned by the service.

CI changes the configuration, asserts exactly one restart, runs the role again, and asserts zero changes and no second restart. Warning suppression was removed, and handler notifications appear in deployment evidence.

Lesson

Ansible success means tasks completed according to their module semantics. It does not automatically prove that the application loaded a changed file. Tie configuration change to a handler, then independently verify effective service state.

The Handler That Never Fired recreates this failure with restart evidence, while Templates That Survive Missing Variables adds safe rendering.