Controller, Nodes, Agents, and Executors

Scenario: A production controller becomes unavailable because compilation ran beside controller services and exhausted memory.

Who actually runs your build?

A very natural first assumption is that "Jenkins" is one single program that both schedules work and runs it. In any real setup, those are two different roles played by two different kinds of machines, and understanding that split is the single most important architectural fact about Jenkins.

Analogy: Think of a delivery company's dispatch office versus its actual delivery vans. The dispatch office (the Jenkins controller) doesn't drive anywhere - it takes incoming orders, decides which van should handle each one based on what that van is equipped for, and keeps records of every delivery ever made. The vans (Jenkins agents) are where the actual driving - the real work - happens. Each van has a certain number of parking spots for packages at once (executors - concurrency slots), and vans get labels like "refrigerated" or "long-haul" so dispatch can match the right job to the right van. If dispatch itself tried to also carry every package in its own back office, the office would grind to a halt the moment two big orders arrived at once.

This is exactly why "don't run your build on the controller" is one of the first rules every Jenkins admin learns: the controller (dispatch office) has to stay responsive to schedule everything, so it should never be the one doing the heavy lifting itself.

Pipeline example

pipeline { agent { label 'linux && nodejs' }; stages { stage('Test') { steps { sh 'npm test' } } } }

Scheduling and execution are deliberately separate

The controller stores configuration and records, evaluates Pipeline orchestration logic, maintains the build queue, and assigns work to available capacity. Nodes are configured compute resources (a VM, a container, a physical machine); agent processes are what actually connect a node to the controller; executors are the concurrency slots on a node - if a node has 4 executors, it can run 4 builds' worth of steps at once. Labels describe capabilities such as operating system, CPU architecture, installed tools, network zone, or specialized hardware (like a GPU). A label expression in a Jenkinsfile (like agent { label 'linux && nodejs' }) is effectively a scheduling API call - changing labels changes which machines can satisfy which jobs, so label changes deserve the same review as any other infrastructure change.

What happens on the "built-in node" - and why it's dangerous

General application builds should never run on the controller's own built-in node. Compilation, running a browser for UI tests, or building container images can exhaust the controller's memory, disk, file descriptors, or process limits - and because the controller is also the thing scheduling every other job, starving it starves the entire Jenkins instance, not just one build. Agents should run with least-privilege identities and constrained network access. Ephemeral agents (spun up fresh per build, then destroyed) give clean, reproducible environments and elastic capacity, but add provisioning and image-download latency to every build's start time. Static, long-lived agents keep warm caches and can host scarce hardware, but need ongoing patching and protection so one build's leftover files or processes can't contaminate the next build that lands on the same machine.

What a scheduling problem looks like in the UI

Build #204 is waiting for next available executor on 'linux && nodejs'
Waiting for next available executor on 'linux && nodejs'

This message means exactly what it says: no node currently online satisfies both labels at once, or every node that does is already fully busy. It is not a sign anything is broken - it's normal queueing - unless it persists far longer than your team's usual build time, which is a capacity or label-mismatch signal worth investigating.

Capacity has more than one dimension

Simply adding more executors only helps if CPU, memory, disk I/O, open network ports, third-party API rate limits, and downstream systems (like an artifact registry) can all actually tolerate the extra concurrency - executors are just slots, not free resources. A fleet of otherwise-idle agents can still produce a long queue if every job requests a label that no online node currently has. Track the age of the oldest item in the queue per label (not just overall), executor utilization, agent provisioning failures, remoting disconnects, workspace disk usage, and resource use of the actual tasks - a single "fleet-wide utilization" percentage can hide a specific starved label entirely.

A pitfall: an agent disappearing mid-deployment

Here's a real incident shape: an ephemeral cloud agent gets reclaimed (spot instance termination, autoscaler scale-down) in the middle of a deployment step. The natural instinct is to wrap the whole pipeline in a generic retry and move on. But whether it's safe to just retry depends entirely on what that step was doing. Re-running a compilation or an isolated test suite is almost always safe - nothing external was changed. Re-running a step that was halfway through a database migration or a partially-completed production release can be dangerous, potentially re-applying a change that already partially landed. The right first move when an agent vanishes mid-step is not "just retry" - it's to check the last durable Pipeline step that completed, check the node's log and its remoting-channel termination message, and check the state of whatever external system that step was talking to, before deciding whether resuming is safe or whether the deployment needs to be reconciled by hand.

Goal: Practice the concrete behavior in jenkins-labeled-parallel-tests. Learn sections have no Jenkins terminal; the Jenkins lab opens the real shared service in a new full-screen tab inside your private folder.