Startup, Shutdown, and Recovery

Recovery is a lifecycle, not only a restart

Recovering correctly from a failure is a whole lifecycle spanning several distinct moments, not just the single act of restarting a crashed process. During startup, the correct sequence is to validate configuration and fully load every required resource before ever marking the instance ready - reversing that order, and marking ready too early, is exactly how the previous section's dishonest-readiness scenario happens. During shutdown, the correct sequence runs in the opposite direction: first mark the instance unready (so the load balancer stops sending it new work), stop accepting brand-new requests, allow whatever work is already in flight a bounded amount of time to finish (a drain period), flush any telemetry that's still safe to write, and only then actually release resources and exit. This whole disciplined sequence is what's meant by graceful shutdown.

A supervisor process, or a container orchestrator sitting above the application, can automatically restart a process after it exits unexpectedly. It's important to add a short delay, or ideally exponential backoff (the same concept from the LiteLLM reliability module earlier in this track, now applied to process restarts instead of network retries), so that a genuinely permanent failure doesn't turn into a tight "crash loop" - restarting instantly over and over, each attempt immediately failing again, consuming CPU and cluttering logs without ever actually fixing anything. It's equally important to cap the total number of restart attempts and clearly surface a truly terminal failure to a human operator, rather than letting the loop spin forever unnoticed.

Warning: Anything held only in an in-memory queue, or chat state kept only in a running process's memory, vanishes the instant that process exits - restarted or not, that in-memory state is simply gone. Genuinely important work needs either idempotent requests (safe to simply retry from scratch, echoing the idempotency concept from the vector-stores module earlier in this track), externally durable storage that survives a process restart, or a protocol explicitly designed to let clients safely retry a request that may or may not have actually completed.

It's also worth clearly distinguishing an unexpected crash from a deliberate, intentional shutdown - a supervisor should normally restart an unexpected nonzero exit code automatically, but should respect and not fight against an operator's deliberately requested stop. Giving each freshly started process instance a fresh, unique identifier is a small but genuinely useful detail: it lets a test, or an operator, concretely prove that a new instance actually started, rather than just assuming a restart happened because some log line implied it.

Tip: Readiness should stay reporting false after any restart until every genuinely required dependency has been confirmed restored - fast restart timing combined with dishonestly-early readiness reporting only makes requests fail sooner and more confusingly, it doesn't actually fix anything about the underlying recovery.