Backup, Restore, and Disaster Recovery

Scenario: Nightly volume snapshots exist, but nobody has restored one or verified encryption keys and external integrations.

If your Jenkins server disappeared right now, could you actually get it back?

This is the single most important question this module asks, and most teams have never genuinely tested the honest answer. It's easy to assume "we have backups" is the same as "we can recover" - but a nightly snapshot that's never been restored is really just an unverified hope, not a tested recovery capability.

Analogy: Think of a fire extinguisher mounted on the wall that's never once been test-fired. It might work perfectly when you finally need it during a real fire - or it might have lost pressure years ago and do nothing at all, and you have no way of knowing which is true until the moment it actually matters most. A backup that's never been restored is exactly the same kind of unverified hope: the presence of a nightly snapshot job running successfully tells you a backup task executed, but it tells you nothing at all about whether that backup can actually be turned back into a working, functioning Jenkins server when you genuinely need it to be.

The entire second half of this module - "restore exercises define recovery," as the deep dive below puts it - exists specifically to turn that unverified hope into an actually tested, trustworthy capability, before a real disaster forces you to find out the hard way.

Pipeline example

// Rebuild layers: pinned controller image, JCasC,
// encrypted JENKINS_HOME backup, external artifacts.

Back up the genuinely irreplaceable state

JENKINS_HOME contains job configurations, build records, plugin state, stored secrets, and the encryption material used to protect those secrets. Omitting the secrets directory or the master encryption key from a backup can leave you with a restore that has credential data present but permanently undecryptable - functionally as useless as having no credentials at all, just with extra confusing steps. Copying files that are actively changing mid-backup (rather than a proper consistent snapshot) can produce internal inconsistency between related files. Because backups genuinely contain live credentials and potentially sensitive log content, encrypt them, restrict who can access them, periodically test their integrity, and apply the same retention discipline to them as you would to any other piece of sensitive production data - a backup is not exempt from the security concerns of the thing it's a backup of.

Separate what's reproducible from what's genuinely irreplaceable

Keep clearly separate layers in your mind: the controller image, Java version, and plugin set should all be pinned and reproducible from version-controlled code (not something that needs "backing up" in the traditional sense, since it can be rebuilt from source). Global configuration and any seed jobs should similarly be managed as code (JCasC). Jenkinsfiles themselves live safely in each application's own SCM repository, entirely independent of the Jenkins controller. Release binaries live in an external artifact repository, also independent of the controller. What's left over - job-specific state, build history, credentials, encryption material - is the genuinely durable state that actually needs a real backup strategy at a consistent point-in-time boundary. Separately, inventory everything external to Jenkins that a full recovery would also depend on: DNS records, TLS certificates, your identity provider configuration, webhook URLs registered on external services, artifact-repository access, and agent connection credentials - a technically-successful Jenkins restore that can't actually reach any of its usual dependencies isn't a real recovery yet.

Restore exercises are what actually define your recovery capability

Restore into a genuinely isolated environment, with all triggers and any deployment-capable network egress deliberately disabled first, so a restored-but-still-connected Jenkins instance can't accidentally start firing real triggers or real deployments while you're just testing the restore process itself. Start up the expected pinned versions and verify: can secrets actually be decrypted, can you log in, are permissions intact, is build history present and browsable, do Pipelines load correctly, do external links resolve, and does a real, simple agent smoke-test build actually succeed end to end.

$ ls /var/jenkins_home_restored/secrets/
master.key  hudson.util.Secret

$ docker exec jenkins-restore-test cat /var/jenkins_home/config.xml | grep version
<version>2.470</version>

Smoke test build #1: SUCCESS (agent connected, credentials decrypted, checkout OK)

That final smoke-test line is the actual proof of recovery - not the presence of the backup files themselves, and not even a clean-looking config file, but a genuine, successful build that exercised credential decryption, agent connectivity, and checkout all together, on the restored instance.

A pitfall: a backup nobody has ever actually restored

Measure your actual recovery point (how much recent work would be lost) and recovery time (how long restoration genuinely takes, start to finish) from a real timed exercise - not from an optimistic guess written into a runbook that's never been tested against reality. Update that runbook based on what you actually learned during the exercise, and make sure to protect or properly rotate any credentials that were used or exposed during the exercise itself. Disaster planning needs to separately consider controller loss, complete agent-fleet loss, and dependency on an external artifact repository being unavailable - these are genuinely different failure modes with different recovery paths, not one single "Jenkins is down" scenario. Only a timed, complete restore that proves identities work, configuration is intact, history is present, and real execution succeeds actually demonstrates recoverability - a nightly snapshot job that reports "success" every night, on its own, only proves that a backup task ran, which is a meaningfully weaker and less reassuring claim than most teams assume it is.

Goal: Practice the concrete behavior in jenkins-scm-trigger-recovery. Learn sections have no Jenkins terminal; the Jenkins lab opens the real shared service in a new full-screen tab inside your private folder.