Monitoring and Evidence-First Incidents

Scenario: The login page is healthy while builds wait an hour behind hung deployments, prompting an evidence-destroying restart.

The dashboard being green doesn't mean nothing's wrong

Here's a genuinely uncomfortable production-Jenkins fact: it's entirely possible for the Jenkins login page to load instantly, look completely healthy, and respond quickly to every request - while every single build in the queue has been silently stuck behind a hung deployment step for the last hour. "Is Jenkins up?" and "is Jenkins actually getting useful work done?" are two very different questions, and only monitoring the first one leaves you blind to the second.

Analogy: Think of a hospital's waiting room. The receptionist's desk being staffed and the phone lines working (the login page loading fine) tells you the front-of-house is operational - but it tells you absolutely nothing about whether patients who checked in an hour ago are actually being seen by a doctor, or are just sitting in the waiting room indefinitely because every doctor is occupied with one unusually long case. Genuinely useful monitoring watches the waiting room itself - how many patients are waiting, and for how long - not just whether the front desk phone still rings.

Everything below is about watching the actual "waiting room" - the build queue, executor utilization, and how long things have genuinely been waiting - rather than just confirming the Jenkins "front desk" is technically open.

Pipeline example

pipeline { agent any; options { timeout(time: 30, unit: 'MINUTES'); timestamps() }; stages { stage('Bounded work') { steps { sh './delivery.sh' } } } }

Observe scheduling and execution, not just uptime

Monitor JVM heap usage and garbage collection behavior, CPU, disk space and inode usage, HTTP response latency, remoting (agent connection) health, how many agents are actually online, executor utilization, overall queue length, and - critically - the age of the oldest item currently sitting in the queue. Segment queue metrics by label specifically, so that (for example) a missing or offline Windows-labeled node doesn't get statistically hidden by plenty of idle, unrelated Linux capacity sitting nearby and dragging the "average" queue time down to something that looks fine overall. Track build duration percentiles (not just the average, which flaky slow outliers can distort), agent provisioning time, checkout time, flaky-test rate, retry counts, and external-system timeouts. Alert on things that genuinely affect users and on clear worsening trends - not on every single metric crossing some arbitrary static threshold regardless of context.

Queue: 14 items waiting
  Oldest item age: 47 minutes (label: 'linux && nodejs')
  Executors online: 8/8 busy
Alert: oldest queued item exceeds 30-minute threshold

This is exactly the kind of alert that a simple "is Jenkins up?" health check would completely miss - Jenkins itself is fully up and responding, every executor is legitimately busy doing real work, and yet real users are still waiting 47 minutes for their build to even start, which is a genuine, actionable problem worth paging someone about.

Bound pipelines so a hang doesn't become a platform-wide failure

Use timeout and concurrency controls throughout your Jenkinsfiles specifically so that one hung delivery pipeline doesn't quietly consume an executor indefinitely and become a platform-wide capacity failure affecting every other unrelated job. Retention policy exists to prevent disk exhaustion on the controller over time - but deliberately deleting logs during an active incident, just to free up space, destroys exactly the evidence you'll need to actually understand what happened; free up space some other way during a live incident, and let retention handle it normally the rest of the time. Correlate a growing queue specifically with executor utilization and the health of whatever external dependency your builds actually rely on (a registry, a database, an API) - reflexively adding more agent capacity during, say, a repository-hosting outage doesn't fix anything; it just means more agents are now also waiting on the same broken external dependency, amplifying the pressure on that already-struggling system instead of relieving it.

Preserve evidence before you restart anything

When responding to an incident, first capture: a timeline of what happened and when, which specific labels and jobs were actually affected, the exact queue reasons Jenkins was reporting, both controller and agent logs, Jenkins' own "support bundle" diagnostic data, disk state, any recent core/plugin/configuration changes that happened around the same time, and the status of relevant external systems. Actively look for the first causal error in that timeline, not just the last, most visible cascading failure that happened to trigger the alert - the alert that paged you is very often three steps downstream of the actual root cause. Only restart Jenkins once you have an actual working hypothesis about what state will be cleared and what currently-running work will genuinely be lost by doing so - restarting reflexively, "just to see if it helps," is exactly how evidence needed to find the real root cause gets destroyed.

A pitfall: the reflexive restart that erases the evidence you needed most

This module's own scenario is the pitfall: "the login page is healthy while builds wait an hour behind hung deployments, prompting an evidence-destroying restart." The actual first move in that exact situation should be to identify what specifically the hung deployment step is waiting on (an external API? A lock that was never released? A crashed downstream process it's still politely waiting to hear back from?) - not to restart Jenkins and hope the problem quietly goes away on its own. Communicate incident scope, relevant build links, any available workaround, an assigned owner, and a committed time for the next status update to stakeholders while you investigate. Once resolved, convert what you found into an actual durable guardrail - a stricter timeout somewhere, a new per-label capacity alert, a plugin-upgrade canary process, a scheduled restore-test, a clearer ownership routing rule, or an updated runbook - so that fast, evidence-based recovery next time depends on having clearly separated controller health, scheduling health, agent health, and the health of external delivery-dependent systems from each other, rather than lumping everything into one undifferentiated "Jenkins is broken" alarm.

Goal: Practice the concrete behavior in jenkins-artifact-promotion. Learn sections have no Jenkins terminal; the Jenkins lab opens the real shared service in a new full-screen tab inside your private folder.