Rotating Secrets and Vault Passphrases

Scenario: An engineer with the production vault passphrase leaves the company. Nobody is sure which credentials they could have read, or how to change the passphrase without breaking every pipeline.

New words, in plain English

• Rotation - replacing a credential or passphrase with a new one.
• ansible-vault rekey - re-encrypts a file with a new passphrase, keeping the contents.
• Compromise assumption - treating a credential as exposed once someone who could read it should no longer be able to.
• Two-key window - a period where both old and new credentials are valid, so rollout can be gradual.

There are two distinct rotations and conflating them causes real confusion.

Rotating the vault passphrase re-encrypts the files with a new key. ansible-vault rekey --new-vault-id prod@prompt file.yml does it, and the contents are unchanged. This protects future access to the ciphertext. It does not change any actual credential - anyone who already decrypted and copied the secrets still has them.

Rotating the secrets themselves changes the actual passwords and tokens in the systems that accept them, then updates the vault. This is what genuinely revokes access, and it is what the scenario requires.

So when someone with the passphrase leaves, both are needed: rekey the vault so they cannot read future ciphertext, and rotate every credential they could have read, because they may already have it.

The practical sequence for a live credential is to create the new one first, add it alongside the old, deploy, verify, and only then revoke the old. That two-key window means the rotation is not itself an outage - which is what makes teams willing to do it often enough to matter.

Analogy: Changing the strongbox lock stops the departing employee opening the box tomorrow. It does nothing about the documents they photographed last week. Both actions are needed, and only one of them is the lock.

A worked example

# 1. Rekey the vault - new passphrase, same contents
ansible-vault rekey --vault-id prod@~/.vault_prod_old \
                    --new-vault-id prod@prompt \
                    group_vars/production/vault.yml

# 2. Rotate the CREDENTIAL itself, with a two-key window
#    a) create the new credential in the target system
#    b) update the vault
ansible-vault edit group_vars/production/vault.yml
#       vault_db_password: "<the new value>"
#    c) deploy and verify
ansible-playbook site.yml --check --diff --limit dbprod
ansible-playbook site.yml --limit dbprod
#    d) verify the application is healthy on the new credential
#    e) ONLY THEN revoke the old credential in the target system

# 3. Update the passphrase wherever automation reads it
#    (CI secret store, deployment pipeline, secret manager)

Automating rotation is what makes it routine rather than an annual fire drill. Where the platform supports it, the strongest arrangement removes long-lived secrets from the repository entirely: a lookup plugin fetches a short-lived credential from a secret manager at run time, so there is nothing in the vault to rotate and nothing in Git to leak.

Vault remains the pragmatic and widely-used answer where that is not available, and it is a genuine improvement over clear text. Just be honest about what it protects: it secures the file at rest, and it does not secure a credential that someone has already read.

Warning: Rekeying the vault does not rotate the secrets inside it. If someone could read the plaintext, assume they did - and change the actual credentials.

Goal: Put this to work in the ansible-vault-secrets lab. Open /labs/ansible, pick ansible-vault-secrets, and fix the real broken project - Ansible really does SSH into four managed hosts and converge them.