Context
External-plane recovery-run ledger, 2026-08-13. Six consecutive restart attempts against the same service over two hours, every one failed identically:
recovery-actuator:embedding-model:restart:2026-08-13T17:51:09.061Z failed executor-failed not-applied
recovery-actuator:embedding-model:restart:2026-08-13T18:21:19.838Z failed executor-failed not-applied
recovery-actuator:embedding-model:restart:2026-08-13T18:39:45.130Z failed executor-failed not-applied
recovery-actuator:embedding-model:restart:2026-08-13T19:07:13.506Z failed executor-failed not-applied
recovery-actuator:embedding-model:restart:2026-08-13T19:36:52.258Z failed executor-failed not-applied
recovery-actuator:embedding-model:restart:2026-08-13T19:55:17.557Z failed executor-failed not-applied
"error": "Docker API POST /containers/<id>/restart?t=10 timed out after 5000ms",
"effectDisposition": "not-applied",
"wallClockMs": 5024
The Problem
Two defects, and the second is the dangerous one.
1. The deadline is shorter than the operation it invokes. The request is restart?t=10 — Docker is explicitly told it may take up to 10 seconds to stop the container gracefully before killing it. The client then abandons the call at 5 seconds. The actuator cannot succeed by construction whenever the container needs more than half its own permitted stop window, which is precisely the saturated-container case recovery exists for.
2. effectDisposition: "not-applied" is asserted from a client timeout. A timed-out HTTP call to the Docker API says nothing about whether the daemon carried out the restart — the request was accepted and the daemon proceeds independently. On the observed plane the container demonstrably did cycle during this window (three distinct engine incarnations in one log tail), while the ledger recorded six consecutive not-applied.
That inversion is worse than a plain failure. Downstream logic treats not-applied as "the effect did not happen, so it is safe to try again", and the actuator re-dispatches every ~20 minutes. If the restarts are landing, recovery is repeatedly restarting a container that was already restarting — destroying in-flight work each time and manufacturing the instability it was invoked to repair.
A timeout is an unknown disposition, not a negative one. The ledger has no way to express that today.
Architectural Reality
t=10 is the daemon-side graceful-stop budget; the client deadline must exceed it plus round-trip, or the call is guaranteed to abandon first.- Container stop latency scales with process state. A container pinned at its CPU limit is the slowest case and the most likely to need recovery — so the timeout is tightest exactly when it must hold.
- Restart is not idempotent with respect to workload: each one discards in-flight requests. Re-dispatching under uncertainty has a real cost, which is what makes a false negative worse than a false positive here.
The Fix (shape)
- Client deadline must exceed the daemon-side stop budget (
t) plus margin, or t must be derived from the deadline. They cannot be chosen independently.
- Introduce an
unknown effect disposition for timeouts and other unacknowledged dispatches. Do not map a client timeout to not-applied.
- On
unknown, re-observe before re-dispatching — the container's startedAt moving is direct evidence the restart landed. The bridge already reads inspect, so the evidence is available without new surface.
- Bound consecutive restart attempts against one target; six identical failures in two hours should escalate rather than continue at cadence.
Acceptance Criteria
Out of Scope
- Why the container was unhealthy (#17063).
- Admission ordering (#17062) and ledger leaks (#17064).
Avoided Traps
- "Just raise the timeout." Necessary but insufficient — without the
unknown disposition, the next slower-than-expected stop reproduces the false negative.
- "Restarts are idempotent." They are not: each discards in-flight work.
Related
- #17063 — the healthcheck driving these restarts
- #17062 · #17064 — same plane, same incident
Context
External-plane recovery-run ledger, 2026-08-13. Six consecutive restart attempts against the same service over two hours, every one failed identically:
"error": "Docker API POST /containers/<id>/restart?t=10 timed out after 5000ms", "effectDisposition": "not-applied", "wallClockMs": 5024The Problem
Two defects, and the second is the dangerous one.
1. The deadline is shorter than the operation it invokes. The request is
restart?t=10— Docker is explicitly told it may take up to 10 seconds to stop the container gracefully before killing it. The client then abandons the call at 5 seconds. The actuator cannot succeed by construction whenever the container needs more than half its own permitted stop window, which is precisely the saturated-container case recovery exists for.2.
effectDisposition: "not-applied"is asserted from a client timeout. A timed-out HTTP call to the Docker API says nothing about whether the daemon carried out the restart — the request was accepted and the daemon proceeds independently. On the observed plane the container demonstrably did cycle during this window (three distinct engine incarnations in one log tail), while the ledger recorded six consecutivenot-applied.That inversion is worse than a plain failure. Downstream logic treats
not-appliedas "the effect did not happen, so it is safe to try again", and the actuator re-dispatches every ~20 minutes. If the restarts are landing, recovery is repeatedly restarting a container that was already restarting — destroying in-flight work each time and manufacturing the instability it was invoked to repair.A timeout is an unknown disposition, not a negative one. The ledger has no way to express that today.
Architectural Reality
t=10is the daemon-side graceful-stop budget; the client deadline must exceed it plus round-trip, or the call is guaranteed to abandon first.The Fix (shape)
t) plus margin, ortmust be derived from the deadline. They cannot be chosen independently.unknowneffect disposition for timeouts and other unacknowledged dispatches. Do not map a client timeout tonot-applied.unknown, re-observe before re-dispatching — the container'sstartedAtmoving is direct evidence the restart landed. The bridge already readsinspect, so the evidence is available without new surface.Acceptance Criteria
tparameter it sends.unknown, nevernot-applied.unknown, the actuator re-observes the target's start time before any re-dispatch, and a moved start time settles the run as applied.unknown/failed runs against one target escalate to a distinguishing state instead of re-dispatching indefinitely.Out of Scope
Avoided Traps
unknowndisposition, the next slower-than-expected stop reproduces the false negative.Related