Proving Handlers Work

Scenario: A post-incident review asks how a change to a config file was applied for six weeks without the service ever being restarted. Nobody had ever verified the handler, only that the run was green.

New words, in plain English

• Evidence - an observable artefact proving something happened - a log line, a timestamp, a process start time.
• Negative test - proving something does NOT happen when it should not.
• Drift injection - deliberately breaking the desired state to confirm automation notices.

A handler is the least-tested part of most playbooks, because the normal path - a converged system - never exercises it. It only runs on the run where something changed, which is exactly the run you are least likely to be watching.

A complete handler test has three parts, and skipping any one of them leaves a real bug undetectable:

1. The positive case. From a clean state, run once. The handler must fire, and there must be evidence - a log line, a new process start time, a marker file.
2. The negative case. Run again immediately. The handler must not fire, and the evidence must not grow. This catches the restart storm.
3. The re-trigger case. Inject drift into the managed file, run again. The handler must fire once more. This catches a handler that silently stopped being notified after a refactor.

That third case is the one people skip, and it is the one that catches a mistyped notify introduced during a later change. A handler that worked when written and quietly stopped working afterwards is indistinguishable from a working one unless you test the re-trigger.

Analogy: Testing a fire alarm means three things: it sounds when there is smoke, it stays quiet when there is none, and it still sounds tomorrow after the electrician has been. Most teams only ever check the middle one, by noticing that it is quiet.

A worked example

# The three-run proof, in order

# 1. Positive - from a clean state the handler must fire
rm -f /var/log/app-reload.log
ansible-playbook site.yml
test -s /var/log/app-reload.log        # evidence exists

# 2. Negative - converged, so nothing should fire
before=$(wc -l < /var/log/app-reload.log)
ansible-playbook site.yml              # must be changed=0
after=$(wc -l < /var/log/app-reload.log)
test "$before" = "$after"               # evidence did NOT grow

# 3. Re-trigger - inject drift, the handler must fire again
echo '# drift' >> /etc/app/app.conf
ansible-playbook site.yml
test "$(wc -l < /var/log/app-reload.log)" -gt "$before"

In real environments the evidence is usually already there: systemctl show app -p ActiveEnterTimestamp gives the exact moment a service last started, and comparing it before and after is a precise, honest test that requires no extra instrumentation.

This pattern generalises well beyond handlers. Positive, negative and re-trigger is the shape of a good test for any conditional automation - and it is exactly the shape the labs in this track use to verify your work.

Tip: Write the negative test first. It is the one that catches the bug you are most likely to ship: automation that does its job and a bit more, every single night.

Goal: Put this to work in the ansible-handler-notify lab. Open /labs/ansible, pick ansible-handler-notify, and fix the real broken project - Ansible really does SSH into four managed hosts and converge them.