Test Failure as a State Transition

Observe the whole sequence

A genuine recovery test needs to walk through the entire sequence end to end, not just confirm the service works before and after some assumed disruption. Start by recording the initial process's identifier, confirming both liveness and readiness are true at the start. Then deliberately remove a required dependency, and check that readiness correctly flips to false while liveness correctly remains true throughout - exactly the distinction the first section of this module was built around. After restoring the dependency, readiness should recover entirely on its own, with no restart needed at all - proving the earlier graceful-lifecycle logic is genuinely dependency-aware rather than only checked once at startup.

Next, trigger a genuinely controlled crash of the process itself. Observe the resulting connection failure, wait a bounded, reasonable amount of time within a fixed deadline, and then require both a different process identifier than before (proof a fresh instance actually started, using the identifier trick from the previous section) and fully healthy probes reporting again. Always make sure to clean up any background test processes at the end, even in the case where one of your assertions along the way actually fails - a failed test that leaves stray processes running behind it just creates a second, unrelated mess to clean up later.

Analogy: A fire drill isn't considered successfully passed simply because a written evacuation plan exists somewhere in a binder. People actually have to exit the building, someone actually has to take attendance and confirm everyone made it out, and normal operations genuinely have to resume afterward. A recovery test plays exactly that same role for a service - it's not passed on paper, it's passed by actually watching every step of the real sequence happen correctly.

A genuinely production-grade test suite for this kind of recovery logic goes further still, covering slow startup sequences, repeated back-to-back crashes, a shutdown that gets stuck partway through, dependency calls that time out rather than failing cleanly, partial capacity loss (some instances down, not all), and concurrent in-flight requests caught mid-flight by a crash. The metrics worth tracking alongside all of this should distinguish clearly between total not-ready duration, restart count, the specific recorded crash reason, and time-to-recovery - lumping all of that into one vague "uptime" number hides exactly the detail an operator needs when something eventually does go wrong.

Goal: Correct the lab's JSON lifecycle policy. Prove dependency-aware readiness that correctly flips both ways, recovery without a restart once the required marker returns, and a genuine automatic FastAPI restart carrying a new process identifier after a simulated crash - the complete sequence described throughout this section, verified end to end.