Docker Registry Incident: Fixing a Broken Private Registry in Production

The Incident

CI pipelines start failing at 09:14 with: error parsing HTTP 400 response body: invalid character '<'. Engineers cannot push images. Pulls of existing images still work. Deployments are blocked.

---

Step 1: Check the Registry Directly

curl -I https://registry.internal/v2/

HTTP/1.1 400 Bad Request
Content-Type: text/html

HTML response to a Docker API call — a load balancer or nginx is intercepting the request and returning an error page before it reaches the registry.

---

Step 2: Check the Registry Container

docker logs registry --tail 50

time="2026-08-20T09:12:34Z" level=warning msg="repository does not exist" go.version=go1.21
time="2026-08-20T09:12:35Z" level=error msg="response completed with error" err="filesystem: mkdir /var/lib/registry/docker/registry/v2/repositories/myapp: no space left on device"

The registry container's volume is full. Writes fail. The load balancer's health check detects the registry returning 500s and starts serving its own error page instead.

---

Step 3: Find the Disk Usage

docker exec registry du -sh /var/lib/registry/docker/registry/v2/

48G /var/lib/registry/docker/registry/v2/

docker exec registry du -sh /var/lib/registry/docker/registry/v2/repositories/*/

12G  /var/lib/registry/.../myapp
 8G  /var/lib/registry/.../frontend
18G  /var/lib/registry/.../ml-model

The ML model image has accumulated 18 GB. The team has been pushing new versions daily without deleting old ones.

---

Step 4: Immediate Fix — Free Space

Identify old tags to delete:

curl -s https://registry.internal/v2/ml-model/tags/list | jq .

{"name":"ml-model","tags":["v1.0","v1.1","v1.2",...,"v8.4","v8.5","v8.6"]}

86 versions, only the last 3 are needed. Delete the rest via the registry API:

for tag in v1.0 v1.1 v1.2 ... v8.3; do
  digest=$(curl -s -I -H "Accept: application/vnd.docker.distribution.manifest.v2+json"     https://registry.internal/v2/ml-model/manifests/$tag | grep Docker-Content-Digest | tr -d '
' | awk '{print $2}')
  curl -X DELETE https://registry.internal/v2/ml-model/manifests/$digest
done

Deleting manifests marks them for removal but does not free disk — you must run garbage collection:

docker exec registry registry garbage-collect /etc/docker/registry/config.yml

This freed 15 GB. The registry came back online within 2 minutes.

---

Step 5: The TLS Certificate Issue

During the investigation, the team checked the cert and found it expires in 6 days:

echo | openssl s_client -connect registry.internal:443 2>/dev/null | openssl x509 -noout -dates

notAfter=Aug 26 00:00:00 2026 GMT

Renewed the Let's Encrypt cert immediately (was handled by certbot on the host):

certbot renew --force-renewal -d registry.internal
nginx -s reload

---

Prevention

1. Automated garbage collection — add a scheduled job:

# /etc/cron.d/registry-gc
0 2 * * * root docker exec registry registry garbage-collect /etc/docker/registry/config.yml >> /var/log/registry-gc.log 2>&1

2. Retention policy in CI — the push pipeline now deletes images older than 30 days:

# In CI, after push
CUTOFF=$(date -d '30 days ago' +%s)
# Delete tags older than cutoff...

3. Disk alerting at 75% on the registry volume.

4. Certificate expiry monitoring — add the registry to the cert monitoring tool.

---

Post-Mortem Summary

Three independent issues converged: unbounded image accumulation (no retention policy), no disk alerting, and a certificate approaching expiry that would have caused a separate outage in 6 days.

Practice Docker registry work in the ShellGenius Docker Labs — the registry-push challenge covers pushing, tagging, and managing images in a private registry.