Building the Two-Run Test Into Delivery

Scenario: A playbook that had passed review for a year was found to be non-idempotent the first time anyone ran it twice in a row - during an incident, when the second run restarted the database.

New words, in plain English

• Regression - a previously working behaviour that breaks due to a later change.
• CI gate - an automated check that blocks a merge when it fails.
• Molecule - the standard test framework for Ansible roles: create an environment, converge, verify, test idempotence, destroy.
• Verifier - the assertion step of a Molecule scenario.

Idempotency is a property that decays. A playbook that is idempotent today acquires a shell task next month, and nothing tells you - because the first run of the day always looks correct.

The cure is a gate rather than a habit. In CI: run the playbook, run it again, and fail the build unless the second run reports changed=0 and failed=0. That is a handful of lines and it catches the entire class permanently.

Molecule formalises this for roles. A Molecule scenario has fixed stages - create, converge, idempotence, verify, destroy - and the idempotence stage is exactly the two-run test, built in. The verify stage runs your own assertions against the converged environment, so a role can assert that a service is genuinely listening rather than merely that a file was written.

Alongside it, ansible-lint catches a large set of problems statically: shell where a module exists, missing changed_when on commands, unnamed tasks, deprecated syntax, unsafe permissions. Running it on every pull request removes a lot of review comments that a machine can make instead.

Analogy: The two-run test is the smoke test in a build pipeline. Every team agrees it matters; only the teams who automated it still have it working a year later.

A worked example

# The whole idempotency gate, in CI
ansible-playbook site.yml
ansible-playbook site.yml | tee second.log
grep -qE 'changed=0.*failed=0' second.log \
  || { echo 'NOT IDEMPOTENT'; exit 1; }

# Static checks on every pull request
ansible-lint
ansible-playbook site.yml --syntax-check

# molecule/default/molecule.yml - the standard scenario
---
driver:
  name: docker
platforms:
  - name: instance
    image: ubuntu:22.04
provisioner:
  name: ansible
verifier:
  name: ansible

# molecule test runs: create -> converge -> idempotence -> verify -> destroy

Molecule's verify stage is where the real value sits, and it is the part teams most often leave empty. Asserting that a config file exists proves very little; asserting that the service is listening on the expected port, that the rendered config parses, and that a health endpoint answers proves the role did its job.

That is the same shape the labs in this track use. Their checkers do not look for files - they run your playbook, run it again to prove idempotency, and inspect the actual end state on the managed hosts over SSH.

Tip: If you add only one CI step to an Ansible repository, make it the second-run check. It costs one extra run and permanently closes the most common class of production surprise.

Goal: Put this to work in the ansible-idempotent-modules lab. Open /labs/ansible, pick ansible-idempotent-modules, and fix the real broken project - Ansible really does SSH into four managed hosts and converge them.