Jenkins Case Study: A Stale Workspace Shipped the Wrong Artifact

The Alert

At 09:32, the Nova customer portal reported release 2026.07.28-rc2 in Jenkins but served UI assets from rc1. The API and database migration were correct. Only the web bundle was stale.

Release build 611 was green, the archive was named portal-611.tgz, and its deployment log said “uploaded successfully.” The team initially suspected the CDN. Purging it changed nothing.

---

Step 1: Compare Identity, Not Filenames

An engineer downloaded the archived artifact and compared its embedded manifest with the Git commit Jenkins claimed to build:

tar -xOf portal-611.tgz manifest.json | jq .gitSha
# "8aa4d91"   (rc1)

git rev-parse HEAD
# f31c0be     (rc2)

sha256sum portal-611.tgz
# 6f8d...  portal-611.tgz

The wrong bytes were already in Jenkins. CDN and deployment transport were cleared.

---

Step 2: Inspect the Workspace History

Build 610 had been manually aborted during frontend compilation. Build 611 reused the same static agent and workspace:

/var/lib/jenkins/workspace/nova-portal_main/
  dist/portal-610.tgz
  dist/manifest.json
  src/

The checkout updated tracked source files but did not remove untracked build output. Git checkout is not workspace cleanup.

The packaging stage contained an overly broad selection:

stage('Package') {
  steps {
    sh 'npm ci && npm run build'
    sh '''
      bundle=$(find dist -name 'portal-*.tgz' | sort | head -1)
      cp "$bundle" "release/portal-$BUILD_NUMBER.tgz"
    '''
    archiveArtifacts artifacts: 'release/*.tgz', fingerprint: true
  }
}

The build produced dist/portal-611.tgz, but lexical sorting selected portal-610.tgz. Renaming it with the current build number made the stale file look current.

---

Step 3: Find Why Cleanup Never Ran

There was a final stage:

stage('Cleanup') {
  steps { cleanWs() }
}

Build 610 was aborted before that stage. A final stage is ordinary work, not a guarantee. Static agents preserved the untracked dist directory for the next run.

The deploy job made provenance worse by rebuilding from the release branch if its input directory was empty. That meant the organization had three possible sources of “the artifact”: build workspace, Jenkins archive, or deployment rebuild.

---

The Immediate Fix

The Pipeline now cleans before checkout and after every outcome, produces one exact path, verifies the embedded SHA, and stashes that artifact:

pipeline {
  agent { label 'node20' }
  stages {
    stage('Checkout') {
      steps {
        deleteDir()
        checkout scm
      }
    }
    stage('Build') {
      steps {
        sh '''
          rm -rf dist release
          mkdir -p release
          npm ci
          npm run build
          test "$(jq -r .gitSha dist/manifest.json)" = "$GIT_COMMIT"
          tar -czf "release/portal-$BUILD_NUMBER.tgz" -C dist .
          sha256sum "release/portal-$BUILD_NUMBER.tgz" > release/SHA256SUMS
        '''
        stash name: 'portal-release', includes: 'release/**'
      }
    }
    stage('Deploy') {
      steps {
        deleteDir()
        unstash 'portal-release'
        sh './deploy.sh "release/portal-$BUILD_NUMBER.tgz"'
      }
    }
  }
  post {
    success { archiveArtifacts artifacts: 'release/**', fingerprint: true }
    always { cleanWs(deleteDirs: true) }
  }
}

The deploy step no longer has permission or code to compile. It must consume the stashed, checksummed output from that run.

---

Longer-Term Controls

The team moved general CI to ephemeral agents, retained a static signing agent only for its hardware-backed key, and gave every build a fresh workspace. Artifact names now include version and Git SHA. A release manifest records source SHA, build URL, checksum, and toolchain version.

They also added disableConcurrentBuilds() to the release job and milestones around promotion so an older run cannot overtake a newer one. Broad find | head artifact discovery is prohibited in release Pipelines; required files are addressed by exact path and tested with test -f.

The root cause was workspace residue. The larger process failure was treating a filename as provenance. An artifact is trustworthy only when its identity is tied from source checkout through test, archive, and deployment.

Practice the repair in Leave a Clean Workspace, Promote the Exact Artifact, and Preserve State for Stage Restart.