LearnNewsExamplesServices
Frontmatter
id17065
titleRecovery actuator restarts a saturated container with a 5s Docker deadline, then records not-applied while the restart proceeds
stateClosed
labels
bugairegressionagent-os
assigneesneo-gpt
createdAtAug 13, 2026, 10:16 PM
updatedAtAug 14, 2026, 2:42 AM
githubUrlhttps://github.com/neomjs/neo/issues/17065
authorneo-opus-vega
commentsCount0
parentIssue17072
subIssues[]
subIssuesCompleted0
subIssuesTotal0
contentTrust
projected
quarantined0
signals[]
blockedBy[]
blocking[]
closedAtAug 14, 2026, 2:42 AM

Recovery actuator restarts a saturated container with a 5s Docker deadline, then records not-applied while the restart proceeds

Closed Backlog/active-chunk-15 bugairegressionagent-os
neo-opus-vega
neo-opus-vega commented on Aug 13, 2026, 10:16 PM

Context

External-plane recovery-run ledger, 2026-08-13. Six restart attempts against the same service over two hours failed identically:

Docker API POST /containers/<id>/restart?t=10 timed out after 5000ms
effectDisposition: not-applied
wallClockMs: 5024

The live container changed incarnation during that interval, while each run claimed the restart was not applied.

The Problem

Two defects compound:

  1. The client deadline is shorter than the operation it invokes. Docker receives restart?t=10, permitting ten seconds for graceful stop, while Neo abandons the call after five seconds. A saturated target can therefore outlive its observer by construction.
  2. A transport timeout is converted into negative evidence. Once request bytes may have reached Docker, losing the response does not prove the restart failed to apply. Recording not-applied authorizes another destructive restart even when the first one may still be executing.

ADR-0026 already provides the correct vocabulary: the effect disposition is uncertain, not a new unknown state. The missing behavior is transport-phase classification plus a durable re-observation interlock that every successor consumes before another POST.

Architectural Reality

  • Docker's t is the daemon-side graceful-stop budget; the HTTP client deadline must exceed it by a validated positive margin.
  • Restart is not workload-idempotent: every applied restart discards in-flight work.
  • StartedAt moving forward is positive evidence that the restart landed. An unchanged or unreadable value is not negative evidence and must remain uncertain.
  • A post-dispatch timeout can coincide with recovery-authority takeover. Therefore the interlock must exist before the POST in authority-fenced durable state, not only in process memory or successor-owned mutable attempt counters.
  • The existing per-window attempt cap remains authoritative. General cross-window futility/freeze policy belongs to #17044.

The Fix (shape)

  1. Validate the Docker grace budget and client margin, then derive clientTimeoutMs = t * 1000 + margin.
  2. Tag every transport failure with its request phase. A timeout, socket reset, aborted response, or equivalent loss after request bytes left Neo records uncertain; a proven pre-dispatch failure may remain not-applied.
  3. Before a compose restart POST, append a recovery-run pending interlock containing the exact container id, StartedAt, deadline, diagnosis, and re-observation request. Authority and live admission are rechecked after this append and immediately before dispatch.
  4. Every later recovery cadence, including a new orchestrator authority holder, consumes the latest run state before normal admission. It inspects rather than POSTing while the effect is unresolved.
  5. Only a positively advanced StartedAt settles the same run as applied. Unchanged/unreadable evidence stays uncertain; a replaced incarnation is recorded as superseded without claiming recovery.

Contract Ledger

Surface Before After
Docker restart deadline independent 5s client / 10s daemon budget validated client deadline strictly greater than t
Transport loss after dispatch not-applied existing uncertain disposition
Durable recovery-run sequence one failed row pendingreobserve-requestedactioned when positively observed
Cross-authority retry guard none latest recovery-run state is consumed before restart admission
Unchanged StartedAt could authorize retry remains deferred + uncertain
Public action/status enums existing values unchanged; no unknown enum added

Acceptance Criteria

  • The Docker restart client deadline is provably greater than the validated t parameter it sends.
  • A post-dispatch timeout or transport loss records uncertain, never not-applied; a proven pre-dispatch failure remains distinguishable.
  • The restart interlock is durable before the POST and survives process recreation plus authority takeover.
  • Before any redispatch, the successor re-observes the exact target incarnation. A moved StartedAt settles the same run as applied; unchanged/unreadable evidence remains uncertain and dispatches no second POST.
  • Authority or live-admission loss during baseline inspection/interlock persistence refuses before the POST.
  • A failed settlement append leaves the uncertain interlock authoritative rather than reopening dispatch.
  • Regression coverage exercises real post-finish socket failure ordering and the timeout-plus-takeover successor path.

Out of Scope

  • Why the provider container was unhealthy (#17063).
  • Tenant admission ordering (#17062) and provider-activity ledger expiry (#17064).
  • General consecutive-failure freeze/escalation across recovery classes (#17044).

Avoided Traps

  • Only raising the timeout. A later slow response would recreate the false negative without transport-phase classification.
  • Treating unchanged observation as failure proof. Absence of a changed timestamp is not proof Docker rejected the request.
  • A process-local latch. It disappears exactly when authority moves or the orchestrator restarts.

Related

  • #17063 — provider-lane healthcheck repair
  • #17062 · #17064 — same external-plane incident
  • #17044 — generalized futility circuit breaker
tobiu referenced in commit 86f3d3d - "fix(ai): reconcile uncertain Docker restarts (#17065) (#17086)" on Aug 14, 2026, 2:42 AM
tobiu closed this issue on Aug 14, 2026, 2:42 AM