Jenkins Incident Case Study: The Green Pipeline That Shipped Failing Tests

The Incident

At 14:08 on a Tuesday, the Atlas team deployed version 4.18.0 of its pricing service. Jenkins showed a green Pipeline: Checkout, Test, Package, and Deploy Production all succeeded. Six minutes later, checkout requests with promotional codes began returning HTTP 500.

The rollback completed at 14:23. The first surprise came from a developer running the release commit locally: 17 integration tests failed. The question was no longer why the application failed. It was why Jenkins had certified it.

---

Step 1: Establish What Jenkins Actually Ran

The console for build 1842 contained failure output:

14:04:31  FAIL test/promotion.integration.test.js
14:04:31    expected 200, received 500
14:04:32  Test Suites: 2 failed, 41 passed, 43 total
14:04:32  Tests:       17 failed, 386 passed, 403 total
14:04:32  writing reports/junit.xml
14:04:32  integration test command completed
14:04:33  [Pipeline] archiveArtifacts
14:04:33  Archiving artifacts
14:04:34  [Pipeline] stage (Package)

The evidence was visible, but Jenkins proceeded. The Test stage had no Tests result link, only an archived XML file. That meant nobody had called the junit Pipeline step.

---

Step 2: Read the Wrapper, Not Just the Jenkinsfile

The Jenkinsfile looked reasonable:

stage('Test') {
  steps {
    sh './ci/run-tests.sh'
    archiveArtifacts artifacts: 'reports/**', allowEmptyArchive: true
  }
}

The defect was in ci/run-tests.sh, changed three days earlier while adding report upload:

#!/usr/bin/env bash
set +e
npm run test:integration -- --reporter=junit
curl -sS -X POST -F report=@reports/junit.xml "$REPORTS_URL"
echo "integration test command completed"

set +e allowed execution after npm returned 1. The script's exit status became the status of the final echo, which was zero. Jenkins' sh step only knows the process exit code; it does not interpret Jest's prose.

The report upload had another flaw. curl lacked --fail, so an HTTP 500 from the report service could also look successful.

---

Step 3: Reproduce the False Green Result

On an isolated agent the team ran:

./ci/run-tests.sh
echo $?

Output:

Test Suites: 2 failed, 41 passed, 43 total
integration test command completed
0

That final zero explained the green stage. Jenkins behaved correctly according to the contract it received.

Git history showed why the change passed review:

git show 7db81ca -- ci/run-tests.sh Jenkinsfile

The author had disabled fail-fast so the upload would still execute after failed tests. The intent was good—retain diagnostics—but failure preservation and evidence publication had been conflated.

---

Step 4: Check the Deployment Gate

Production deployment used only the branch name:

stage('Deploy Production') {
  when { branch 'main' }
  steps { sh './ci/deploy.sh production' }
}

There was no separate quality-gate state. Once Test returned zero, deployment was inevitable. The archived JUnit XML was inert: archiving a file does not parse tests or change build health.

---

The Fix

The wrapper now captures and restores the test exit code while allowing evidence upload:

#!/usr/bin/env bash
set -uo pipefail

test_status=0
npm run test:integration -- --reporter=junit || test_status=$?

if [[ -f reports/junit.xml ]]; then
  curl --fail --silent --show-error     -X POST -F report=@reports/junit.xml "$REPORTS_URL" || true
fi

exit "$test_status"

The Jenkinsfile publishes results in a post condition, so publication still runs after the shell step fails:

stage('Test') {
  steps {
    sh './ci/run-tests.sh'
  }
  post {
    always {
      junit testResults: 'reports/junit.xml', allowEmptyResults: false
      archiveArtifacts artifacts: 'reports/**', allowEmptyArchive: true
    }
  }
}

The optional external upload may warn, but it cannot erase the test command's status. Missing JUnit output is now itself an error rather than silently accepted.

---

Prevention Added After the Incident

• A fixture test intentionally fails the wrapper and asserts a non-zero exit.
• Shell scripts use set -euo pipefail unless exit handling is explicit.
• Reviewers reject || true and returnStatus: true without a documented result policy.
• Required reports use allowEmptyResults: false.
• Deployment consumes an artifact produced only after required gates pass.
• The team alerts on a sharp drop to zero reported tests.

The outage lasted 15 minutes; the semantic defect had existed for three days. Green means only that every enforced gate passed. The engineering task is making sure the right gates are enforced.

Recreate the relevant mechanics in Publish Test Results, Enforce a Quality Gate, and Contain a Non-Critical Failure.