Docker Debugging Case Study: Tracing a Container Crash in Production

The Alert

It is 11:47 PM. You get paged: the checkout frontend is returning 502s. The Nginx container serving the React app has crashed three times in the last hour. Auto-restart brought it back each time but response times are spiking.

This walkthrough traces exactly how you would debug this, command by command.

---

Step 1: Establish What Is Running

First, get the lay of the land. What containers are running, and what state are they in?

docker ps -a --format "table {{.Names}}	{{.Status}}	{{.Image}}	{{.Ports}}"

Output:

NAMES             STATUS                     IMAGE          PORTS
checkout-nginx    Up 4 minutes               nginx:alpine   0.0.0.0:443->443/tcp
checkout-app      Up 3 hours                 node:18
db-primary        Up 2 days                  postgres:15

The nginx container is only 4 minutes old. The app and database have been stable. That narrows it to Nginx or something Nginx depends on.

---

Step 2: Check the Logs

docker logs checkout-nginx --tail 100

You see this near the bottom of the previous run:

2026/08/19 23:44:12 [crit] 7#7: *512 connect() to unix:/var/run/php-fpm.sock failed (2: No such file or directory)
2026/08/19 23:44:18 [warn] 7#7: *514 client_max_body_size exceeded, client: 10.0.0.45
Killed

The last line - Killed - is the kernel OOM killer terminating the process. The container was not crashing due to a configuration error. It ran out of memory.

The PHP-FPM socket error is a red herring from an old config that was never cleaned up. Focus on the OOM kill.

---

Step 3: Confirm Memory Usage

docker stats checkout-nginx --no-stream

Output:

NAME              CPU %     MEM USAGE / LIMIT     MEM %     NET I/O         BLOCK I/O
checkout-nginx    2.1%      247MiB / 256MiB       96.5%     1.2GB / 890MB   12.3MB / 0B

The container is at 96.5% of its 256 MB limit. Under normal traffic it hovers around 200 MB, but a traffic spike pushed it over. The container crashed and restarted, temporarily dropping memory usage, but the next spike will do the same thing.

---

Step 4: Inspect the Container Configuration

docker inspect checkout-nginx | grep -A5 '"Memory"'

Output:

"Memory": 268435456,
"MemorySwap": 268435456,
"MemoryReservation": 0,

268435456 bytes = 256 MB. Memory swap equals memory limit, meaning there is zero swap space allocated. When the container hits 256 MB, the OOM killer fires immediately with no buffer.

Let's also check how the container was started to understand who set this limit:

docker inspect checkout-nginx --format='{{.HostConfig.Memory}}'
# 268435456

The limit was set at container creation via --memory 256m.

---

Step 5: Look Inside the Running Container

docker exec -it checkout-nginx /bin/sh

Inside the container:

nginx -T | grep worker_processes
# worker_processes auto;

nginx -T | grep worker_connections
# worker_connections 1024;

cat /proc/meminfo | grep -E "MemTotal|MemFree|MemAvailable"

Output:

MemTotal:        262144 kB
MemFree:          12288 kB
MemAvailable:     18432 kB

worker_processes auto means Nginx is spawning one worker per CPU core it can see. Because this container can see all host CPUs (Docker does not isolate CPU count by default), it may be spawning 8 or 16 workers when the machine only has a few cores available to it.

Each worker maintains connection buffers. 1024 connections x multiple workers at 256 MB total = a crash waiting to happen.

---

Step 6: Check the Logs Volume

docker exec checkout-nginx du -sh /var/log/nginx/
# 1.1G /var/log/nginx/

There is 1.1 GB of logs inside the container. The access log alone is growing at roughly 500 MB per day, with no rotation. On top of the worker memory issue, the container is holding over a gigabyte of log data in its writeable layer.

---

The Resolution

The fix has two parts.

Part 1: Memory and worker configuration

Update the Nginx config to pin worker count:

worker_processes 2;
worker_connections 512;

Increase the container memory limit and add swap headroom in the docker-compose.yml:

services:
  checkout-nginx:
    image: nginx:alpine
    mem_limit: 512m
    memswap_limit: 768m

Part 2: Log rotation with the json-file driver

services:
  checkout-nginx:
    image: nginx:alpine
    logging:
      driver: json-file
      options:
        max-size: "20m"
        max-file: "5"

This caps log storage at 100 MB total (5 files x 20 MB) and rotates automatically.

---

What the Post-Mortem Documented

Three issues contributed to this incident:

Memory limit set too low during initial deployment, not reviewed when traffic grew.
worker_processes auto in Nginx config caused more workers than needed, each consuming buffer memory.
No log rotation - log files growing unbounded inside the container were consuming the writeable layer and eventually contributed to memory pressure when Nginx tried to flush buffers to disk.

None of these are exotic. All three are in the top 10 Docker production mistakes teams make. The debugging took 20 minutes; the fix took 5.

If you want to practice this kind of debugging in a safe environment, the ShellGenius Docker Labs have 30 container incident scenarios including OOM debugging, log forensics, and resource limit tuning - all in a real Docker daemon in the browser.