Core, Plugins, and Upgrade Strategy

Scenario: Dozens of plugins are updated directly before a critical release and a resumable Pipeline breaks.

Jenkins isn't one program - it's a core plus dozens of moving parts

A newcomer often pictures "upgrading Jenkins" as swapping one monolithic application for a newer version, the way you'd update a phone app. In reality, a typical Jenkins instance is a relatively small core, plus dozens of independently-versioned plugins, each with their own dependencies on each other and on specific core versions - and "just click update" on all of them at once is exactly how production Jenkins outages happen.

Analogy: Think of Jenkins less like a single sealed appliance and more like a large, interconnected Lego structure built from many different specialized kits, where some pieces from the "credentials" kit are load-bearing supports for pieces from the "Pipeline" kit sitting on top of them. Swapping out one kit's pieces for a newer version of that kit, without checking whether the pieces it connects to still fit the same way, risks the whole structure shifting or collapsing somewhere you weren't even looking - possibly in a completely different part of the structure than the piece you actually changed.

Everything in this module's "canary" and "staging first" advice exists because of this one structural fact: plugins form a real dependency graph, and updating one node in that graph can genuinely ripple out to behavior in a completely different, seemingly unrelated part of your Jenkins instance.

Pipeline example

pipeline { agent any; stages { stage('Upgrade smoke suite') { steps { build job: 'pipeline-smoke-tests' } } } }

Upgrades change a dependency graph, not just one component

Plugins declare minimum required core and plugin versions, and optional integrations between plugins add further dependency edges on top of that. Updating one visible plugin (say, a new SCM integration) can transitively alter how credentials, source-control checkout, Pipeline execution, or even internal serialization behavior works - completely outside the area you thought you were touching. Before upgrading anything, export the current installed plugin versions as a baseline, actually read the upgrade guides and security advisories (not just the changelog headline), inspect what dependency versions are changing underneath the plugin you're focused on, and remove plugins that are genuinely unused - every installed plugin adds code, configuration surface, startup time, and its own compatibility constraints to maintain going forward, whether or not anyone's actively using it.

Installing plugin: workflow-cps 2.95 -> 3.12
  requires: workflow-step-api >= 2.24 (currently 2.22, will upgrade)
  requires: script-security >= 1.78 (currently 1.78, OK)
Restart required to complete installation

Notice this log surfaces a transitive requirement - upgrading workflow-cps (the core Pipeline execution engine) also forces an upgrade of workflow-step-api underneath it, which is exactly the kind of ripple effect that makes "just update this one plugin" riskier than it sounds.

Test in staging with the same configuration as production

Build a staging environment from the exact same pinned controller configuration and representative restored jobs as production - not a bare-bones test instance with none of the real complexity. Test authentication, RBAC, agent connectivity, webhooks, credential binding, Pipeline job creation, a paused input step, restart-and-resume behavior, JUnit publishing, artifact archiving, and outbound notifications - specifically because a healthy-looking login page proves absolutely nothing about whether any of these deeper behaviors still work correctly after the upgrade. Back up everything before migrating, and explicitly account for the fact that some plugins change their own stored data format on upgrade, which matters directly for your rollback plan if you need to revert.

Canary the rollout with real operational evidence, not just a clean install

Pin the controller's container image, Java version, core version, and every plugin version together in reviewable, version-controlled code - not as a set of manual clicks in the UI that nobody wrote down. Drain or explicitly account for currently-running work before upgrading, avoid upgrading right before a known critical release window, and capture baseline queue and executor metrics before the change so you have something concrete to compare against afterward. After rolling out, actively watch for startup errors, Configuration-as-Code (JCasC) validation rejections, remoting/agent-connection issues, unusual queue reasons, Pipeline-specific exceptions, and login failures - not just "did the homepage load."

A pitfall: an urgent security patch applied to everything, untested, right before a big release

Keep a previously-known-good, tested controller image and a documented restore procedure ready at all times, not just when a big upgrade is already planned. An urgent security fix can genuinely shorten how much time you have available for the usual staging process - but it does not remove the actual need to understand exactly what's exposed, exactly which permissions or plugins are affected, what a reasonable smoke-test looks like even under time pressure, and what your specific rollback criteria are if something goes wrong. The real incident this scenario describes - "dozens of plugins updated directly right before a critical release, and a resumable Pipeline breaks" - happens precisely when that discipline gets skipped under time pressure, and it tends to happen at exactly the worst possible moment: right when you most needed everything to keep working smoothly.

Goal: Practice the concrete behavior in jenkins-declarative-rescue. Learn sections have no Jenkins terminal; the Jenkins lab opens the real shared service in a new full-screen tab inside your private folder.