Test Failure as a State Transition

Observe the whole sequence

A recovery test records the initial process identifier, confirms liveness and readiness, removes a dependency, and checks that readiness becomes false while liveness remains true. After restoring the dependency, readiness should recover without a restart.

Next, trigger a controlled crash. Observe a connection failure, wait within a bounded deadline, and require a different process identifier plus healthy probes. Always clean up background processes, even when an assertion fails.

Analogy: A fire drill is not passed because an evacuation plan exists in a binder. People must exit, accountability must work, and normal operation must be restored.

Production tests also cover slow startup, repeated crashes, stuck shutdown, dependency timeouts, partial capacity, and concurrent in-flight requests. Metrics should distinguish not-ready duration, restart count, crash reason, and recovery time.

Goal: Correct the lab's JSON lifecycle policy. Prove dependency-aware readiness, recovery without restart when the marker returns, and automatic FastAPI restart with a new process identifier after a simulated crash.