Context
External-plane recovery-run ledger, 2026-08-13. Six restart attempts against the same service over two hours failed identically:
Docker API POST /containers/<id>/restart?t=10 timed out after 5000ms
effectDisposition: not-applied
wallClockMs: 5024
The live container changed incarnation during that interval, while each run claimed the restart was not applied.
The Problem
Two defects compound:
- The client deadline is shorter than the operation it invokes. Docker receives
restart?t=10, permitting ten seconds for graceful stop, while Neo abandons the call after five seconds. A saturated target can therefore outlive its observer by construction.
- A transport timeout is converted into negative evidence. Once request bytes may have reached Docker, losing the response does not prove the restart failed to apply. Recording
not-applied authorizes another destructive restart even when the first one may still be executing.
ADR-0026 already provides the correct vocabulary: the effect disposition is uncertain, not a new unknown state. The missing behavior is transport-phase classification plus a durable re-observation interlock that every successor consumes before another POST.
Architectural Reality
- Docker's
t is the daemon-side graceful-stop budget; the HTTP client deadline must exceed it by a validated positive margin.
- Restart is not workload-idempotent: every applied restart discards in-flight work.
StartedAt moving forward is positive evidence that the restart landed. An unchanged or unreadable value is not negative evidence and must remain uncertain.
- A post-dispatch timeout can coincide with recovery-authority takeover. Therefore the interlock must exist before the POST in authority-fenced durable state, not only in process memory or successor-owned mutable attempt counters.
- The existing per-window attempt cap remains authoritative. General cross-window futility/freeze policy belongs to #17044.
The Fix (shape)
- Validate the Docker grace budget and client margin, then derive
clientTimeoutMs = t * 1000 + margin.
- Tag every transport failure with its request phase. A timeout, socket reset, aborted response, or equivalent loss after request bytes left Neo records
uncertain; a proven pre-dispatch failure may remain not-applied.
- Before a compose restart POST, append a recovery-run
pending interlock containing the exact container id, StartedAt, deadline, diagnosis, and re-observation request. Authority and live admission are rechecked after this append and immediately before dispatch.
- Every later recovery cadence, including a new orchestrator authority holder, consumes the latest run state before normal admission. It inspects rather than POSTing while the effect is unresolved.
- Only a positively advanced
StartedAt settles the same run as applied. Unchanged/unreadable evidence stays uncertain; a replaced incarnation is recorded as superseded without claiming recovery.
Contract Ledger
| Surface |
Before |
After |
| Docker restart deadline |
independent 5s client / 10s daemon budget |
validated client deadline strictly greater than t |
| Transport loss after dispatch |
not-applied |
existing uncertain disposition |
| Durable recovery-run sequence |
one failed row |
pending → reobserve-requested → actioned when positively observed |
| Cross-authority retry guard |
none |
latest recovery-run state is consumed before restart admission |
Unchanged StartedAt |
could authorize retry |
remains deferred + uncertain |
| Public action/status enums |
existing values |
unchanged; no unknown enum added |
Acceptance Criteria
Out of Scope
- Why the provider container was unhealthy (#17063).
- Tenant admission ordering (#17062) and provider-activity ledger expiry (#17064).
- General consecutive-failure freeze/escalation across recovery classes (#17044).
Avoided Traps
- Only raising the timeout. A later slow response would recreate the false negative without transport-phase classification.
- Treating unchanged observation as failure proof. Absence of a changed timestamp is not proof Docker rejected the request.
- A process-local latch. It disappears exactly when authority moves or the orchestrator restarts.
Related
- #17063 — provider-lane healthcheck repair
- #17062 · #17064 — same external-plane incident
- #17044 — generalized futility circuit breaker
Context
External-plane recovery-run ledger, 2026-08-13. Six restart attempts against the same service over two hours failed identically:
The live container changed incarnation during that interval, while each run claimed the restart was not applied.
The Problem
Two defects compound:
restart?t=10, permitting ten seconds for graceful stop, while Neo abandons the call after five seconds. A saturated target can therefore outlive its observer by construction.not-appliedauthorizes another destructive restart even when the first one may still be executing.ADR-0026 already provides the correct vocabulary: the effect disposition is
uncertain, not a newunknownstate. The missing behavior is transport-phase classification plus a durable re-observation interlock that every successor consumes before another POST.Architectural Reality
tis the daemon-side graceful-stop budget; the HTTP client deadline must exceed it by a validated positive margin.StartedAtmoving forward is positive evidence that the restart landed. An unchanged or unreadable value is not negative evidence and must remain uncertain.The Fix (shape)
clientTimeoutMs = t * 1000 + margin.uncertain; a proven pre-dispatch failure may remainnot-applied.pendinginterlock containing the exact container id,StartedAt, deadline, diagnosis, and re-observation request. Authority and live admission are rechecked after this append and immediately before dispatch.StartedAtsettles the same run as applied. Unchanged/unreadable evidence stays uncertain; a replaced incarnation is recorded as superseded without claiming recovery.Contract Ledger
tnot-applieduncertaindispositionpending→reobserve-requested→actionedwhen positively observedStartedAtdeferred+uncertainunknownenum addedAcceptance Criteria
tparameter it sends.uncertain, nevernot-applied; a proven pre-dispatch failure remains distinguishable.StartedAtsettles the same run as applied; unchanged/unreadable evidence remains uncertain and dispatches no second POST.Out of Scope
Avoided Traps
Related