An unhealthy container is diagnosed critical and never acted on

Context

An external Agent OS plane was observed at 2026-08-09T01:26:19Z with its Memory Core container unhealthy while its process was alive — --max-old-space-size=768 …memory-core/mcp-server.mjs, 710 MB resident, 0.9% CPU — beside a healthy Knowledge Base container. It had been in that state long enough for both remote seats to lose Memory Core, and nothing on the Neo side would have changed that at any point.

This ticket is not about why Memory Core wedges — that is #16677, and its mechanism is deliberately unclassified. This is about what happens after the wedge is correctly diagnosed: nothing.

Observation and inference are separated below. The observations are from live source and a live deployment snapshot; the inference is only that the missing link is a controller.

The Problem

ADR-0026 §2.4 specifies that phase-1 ships a reactive controller — diagnosis-class → fixed action — sitting between the diagnosis layer and the actuator. It has not shipped for the container-lifecycle world. The consequence is a complete detect half and a complete actuator half with no path between them:

observe → classify → RECORD → (stop)

Measured on dev and on a live plane:

• container-unhealthy is a first-class diagnosed fact — severity: 'critical', authoritative: true when the health state is unhealthy (ContainerHealthDiagnosisService:503-511). The detect half works. Corrected 2026-08-09, before any implementation: the detect half does NOT work, and this was the load-bearing error in the original framing. The fact is created; a classification is not. classifyFacts:774 routes the unhealthy fact through hasAuthoritativeEvidence, which at :903 requires countAuthoritativeFacts(allFacts) >= minAuthoritativeFacts — 2, and a quiet unhealthy-but-alive container produces exactly 1. Its only escape hatch is an endpointProbeFailed fact, and that hatch is dead in production: grep -rn "endpointProbe" ai/daemons/orchestrator/ --include='*.mjs' returns zero callers outside the diagnosis service's own definition, and there is no endpointProbe leaf in ai/configBase.mjs, so collectEndpointProbeFacts always receives undefined and returns []. It is additionally authoritative: false (:604-618), so it could never satisfy the >= 2 floor either — the OR-branch was the only way it ever mattered. The classification therefore falls through to classifyFacts:891's return null, and diagnose:305-314 emits status: 'advisory' with diagnosis: null and actionClass: null.

  Why this matters more than the gap the ticket was filed for: wire the perfect consumer and this failure mode still heals nothing, because the decision it consumes carries no action class. Shipping only the controller would have produced a green PR, a genuinely new code path, and zero behaviour change on the case that motivated the ticket — detectable only by re-deriving this chain. The ticket's conclusion survives; its mechanism did not.

  Stated in the less alarming direction too, since I checked for the worse version and did not find it: I expected status: 'healthy' here — the green-on-an-unmeasured-axis shape this daemon has been bitten by before. It is not. diagnose:311 returns 'advisory' whenever any fact exists. The service is surfaced; it simply carries nothing actionable.

• Restart policies exist and are the wrong instrument — corrected 2026-08-09. An earlier revision of this bullet claimed no compose service declares a restart: policy. That was wrong: I grepped only the base docker-compose.yml, and the policies live in the overlay layer that actually runs. docker-compose.local-agent-os.yml declares restart: unless-stopped on five services, and the observed external plane declares it on six. The claim that survives is stronger: Docker's restart: policy is exit-driven, not health-driven — it never fires for a container the runtime considers unhealthy while its process is still alive. The policies were present on the observed plane and Memory Core stayed down anyway.

• The heal-event ledger is empty on a plane with ~9 h uptime: selfHeal: {total: 0, byStatus: {}, recentEvents: [], lastEventAt: null} from the deployment-state bridge snapshot.

• Every recovery-run entry ever written is action: 'record', recoveryClass: 'ambiguous', status: 'recorded' — and all of them target the backup supervised task, never a compose service.

• Every prerequisite was armed and still nothing acted — verified against the observed deployment's own composition, 2026-08-09. This is the section that makes the ticket falsifiable rather than argued, so it is stated as configuration rather than inference:

  prerequisite	state on that plane
  runtime-access mechanism	NEO_ORCHESTRATOR_RUNTIME_ACCESS_MECHANISM=docker-socket
  socket reachable	/var/run/docker.sock bind-mounted into the orchestrator
  target allowlisted	the wedged service is named in NEO_ORCHESTRATOR_RUNTIME_ACCESS_ALLOWED_SERVICES
  restart operation granted	NEO_ORCHESTRATOR_RUNTIME_ACCESS_LIFECYCLE_OPERATIONS=restart
  actuator enabled	NEO_RECOVERY_ACTUATOR_ENABLED absent ⇒ default, and the default is enabled (ADR-0026 §2.2 / #13952)
  runtime restart policy	restart: unless-stopped on six services
  third-party healer	none — no autoheal, watchtower, or restart sidecar in the composition

  Socket mounted, service allowlisted, restart granted, actuator on by default, restart policy declared — and the unhealthy service was never restarted. That eliminates missing-mechanism, missing-privilege, missing-allowlist, missing-actuator, disabled-actuator, missing-restart-policy and external-healer as explanations, by configuration. One explanation survives: nothing consumes the decision.

So an authoritative critical fact about an unreachable service produces a durable record that no code path consumes.

The failure shape this creates is the one ADR-0026 exists to prevent — not a wrong action, but an indefinite non-action that is indistinguishable from a healthy system to anything except a human reading a dashboard. #16677's AC-4 already states the invariant this violates: "the server either remains responsive with backpressure/degraded errors, or fails in a way the deployment can detect and recover. A permanently live-but-unreachable process is not an accepted state."

The Architectural Reality

The pattern already exists in the same daemon, one world over. This is the load-bearing finding, and it makes the gap a wiring omission rather than a design question:

Both actuators are constructed by the same Orchestrator. The lifecycle actuator's apply(serviceKey, action) interface is shipped, admission-gated (isActionAllowedForTarget), and anti-thrash-enveloped. It is reachable and never reached from a health diagnosis.

Prior art naming the same gap without owning it: #16636 records "no controller consumes container-health decisions yet, so 'routes to X' means prescribed in the recorded diagnosis, never actuated", and scopes the controller out as a design lane rather than a leaf. #16596 carries the same note. Neither is a filed deliverable, which is why this ticket exists.

The Fix

Direction, not premature mechanism — the same posture #16677 takes.

Two parts, one deliverable — revised 2026-08-09 after tracing the classification path. Neither half heals the observed failure alone: part 1 without part 2 produces a diagnosis nothing acts on; part 2 without part 1 produces a consumer with nothing to read. They ship together or the ticket closes on a behaviour change that did not happen.

0. Supply the direct endpoint probe, so ADR-0025 §2.4's sanctioned pair can form. RETRACTED-AND-REPLACED 2026-08-09. The original item asked to let one authoritative container-unhealthy fact classify alone, justified by analogy to the store-memory carve-out. That analogy fails twice: the store carve-out was sanctioned as an explicit ADR amendment (#16596) and it carries a measured sustained window, while this had neither and was filed as "amends nothing". The deeper error is the analogy itself — debouncing answers NOISE and cannot answer CONTRADICTION, because repeated evaluations of one probe are still one evidence channel and that channel can be measuring the wrong thing. ADR-0025 §2.1 names the live instance: a provider-dependent canary false-fails while the service still answers and persists, so restarting it is a self-inflicted outage. What replaces it: ADR-0025 §2.4's pair — container-unhealthy plus a failed DIRECT endpoint probe — is already admitted by hasAuthoritativeEvidence's endpointProbeFailed OR-branch. It has always worked. It is unreachable in production for exactly one reason: nothing supplies endpointProbe, which this ticket's own Context section documented ("zero callers outside the diagnosis service's own definition") while still concluding the floor was the defect. The repair is therefore a PRODUCER — a direct service probe wired at the bridge, reusing runHealthcheck's existing expectedStatus semantics so a degraded service is not counted as a failed probe. No classifier change, no ADR amendment. This is the remaining scope of this ticket; PR #16778 delivers the consumer half.
1. Wire a reactive controller for the container-lifecycle world, mirroring DataIntegrityDiagnosisService's shape: consume the container-health decision, map diagnosis-class → action, call RecoveryActuatorService.apply inside the existing §2.5 envelope. The actuator, its admission matrix, and its anti-thrash state all already exist; this adds the consumer.
2. Decide the container-unhealthy mapping explicitly. restart is admitted for compose services today. Whether an unhealthy-but-alive service should be restarted, and after how many consecutive authoritative observations, is the design content — not whether a path should exist.
3. Pair it with the terminal posture #16677 AC-4 requires. A controller that restarts a wedged service and a service that self-terminates when its own surface is unreachable are complementary; either alone leaves a hole, and the second needs a restart: policy to be useful.
4. Do not widen the action set. ADR-0026 AC-9 forbids widening the closed action set via an implementation sub; this ticket adds a consumer of existing actions only.

Contract Ledger Matrix

Decision Record impact

aligned-with ADR 0026 — §2.4 already specifies the reactive controller as phase-1 and fixes the interface it must use. This implements sanctioned-but-unbuilt scope; it amends nothing, widens no action set, and grants no new privilege. If the chosen mapping turns out to require a new action class, that is a separate amendment and must be sanctioned in the ADR in the same change that ships it (AC-9).

Acceptance Criteria

• An authoritative container-unhealthy fact, alone, produces a diagnosed decision carrying a non-null actionClass. RETRACTED 2026-08-09, and it was the load-bearing error in this ticket's prescription. This AC contradicted the false-unhealthy AC below — both cannot hold — and I resolved that contradiction in the unsafe direction when implementing it. @neo-gpt falsified the implementation at PR #16778's exact head: State=running + Health=unhealthy + endpointProbe.ok=true still returned restart, because collectEndpointProbeFacts discards any probe whose ok !== false. ADR-0025 §2.4 requires the PAIR — container-unhealthy plus a failed direct endpoint probe — and hasAuthoritativeEvidence already admits exactly that pair via its endpointProbeFailed OR-branch. Resolution: withdrawn, not delivered. The retraction above IS the disposition — the prescription was wrong and the safe pair (container-unhealthy + a failed direct probe) is what shipped. Receipt: ContainerHealthControllerService.spec.mjs — SAFETY — an ANSWERING service is not restarted even with a SECOND authoritative fact. Replaced by: the ADR-sanctioned pair becomes REACHABLE IN PRODUCTION. The classifier needs no change at all; the defect is that nothing supplies endpointProbe, so the pair never forms. That is a missing producer, not a floor that is too strict — and this ticket's own Context section stated the zero-caller evidence while still concluding the floor was wrong.
• Part 0 must not over-generalise, and this is the first thing a reviewer should attack. A starting health state produces a severity: 'warning', authoritative: false fact (:503-511) and must still not classify. The change is scoped to the runtime's debounced unhealthy verdict, never to "any non-healthy health state" — the latter would act during every container's normal startup window, which is the obvious wrong version of this fix. Receipt: ContainerHealthControllerService.spec.mjs — NEGATIVE — a STARTING container is not actioned, so an ordinary boot window is never healed.
• A container-health decision carrying an authoritative container-unhealthy fact reaches RecoveryActuatorService.apply through a named controller, within the existing §2.5 envelope. Receipt: ContainerHealthControllerService.spec.mjs — a genuinely wedged container reaches the actuator and is restarted, driving the real RecoveryActuatorService through ContainerHealthControllerService.
• throttleShed maps to no admitted action and must route to record-with-diagnosis, enumerated rather than silently defaulted. The actuator's closed set is reconfigure / restart / redeploy / warm-provider / raise-ceiling; exhaustion-non-store and contention both emit throttleShed, which is in none of them. Inventing one here is exactly what AC-9 forbids. That gap is real and is someone's ticket; it is not this one's licence. Receipt: ContainerHealthControllerService.spec.mjs — throttle-shed maps to no admitted action and records rather than inventing one.
• The diagnosis-class → action mapping is explicit and enumerated; an unmapped class records-with-diagnosis exactly as today rather than defaulting to any action. Receipt: CONTAINER_HEALTH_ACTION_ROUTES is frozen and total over CONTAINER_HEALTH_ACTION_CLASSES; ContainerHealthControllerService.spec.mjs — every diagnosed action class has a route — a new class fails HERE, not in production and FAIL-CLOSED — an action class with no route records rather than inheriting an action.
• The action set is provably unwidened: a spec asserts an action outside the target kind's admitted row is still refused after this change. Receipt: ContainerHealthControllerService.spec.mjs — the action set is provably unwidened — every routed action is one the actuator already admits, plus the out-of-kind refusal control.
• The anti-thrash envelope is proven to bind the new path — a repeating unhealthy fact reaches alarm-only rather than an unbounded restart loop, asserted rather than argued. Receipt: ContainerHealthControllerService.spec.mjs — a repeating unhealthy fact marches into alarm-only instead of restart-looping.
• Every controller decision writes a heal-event, including the decision not to act, so selfHeal.total: 0 can no longer mean either "nothing broke" or "nothing was consumed". Receipt: ContainerHealthControllerService.spec.mjs — EVERY consumed decision writes exactly one heal-event, including the decision not to act, and the decision is recorded to the ledger the deployment snapshot actually folds (the ledger-path convergence, since the prior sink had no production reader).
• Negative control: a healthy service produces no action, and a diagnosis whose class has no mapping produces a record and no action. A test asserting only that a restart fired would pass against a controller that restarts everything. Receipt: ContainerHealthControllerService.spec.mjs — NEGATIVE — a healthy container produces no action and no ledger entry, plus the FAIL-CLOSED unmapped-class control.
• False-unhealthy control — the one this design is most likely to get wrong. A service that is answering while its probe reports unhealthy must not be restarted by the first authoritative fact. The controller keys on the debounced container state (the runtime's own retries × interval window), never on a raw probe result, and the sustained threshold is stated as a number rather than inherited implicitly. See the Avoided Traps entry — this failure mode is documented, not hypothetical. Receipt: ContainerHealthControllerService.spec.mjs — SAFETY — a service ANSWERING while the runtime reports unhealthy is never restarted. The missing producer was supplied: DeploymentStateBridgeService.collectDirectProbe + the exported classifyDirectProbeOutcome, so ADR-0025 §2.4's pair can actually form.
• Coverage fails against the current unwired state and passes against the repair. Receipt: every repair on PR #16778 is mutation-convicted in both directions, each checked to redden its expected control — the gate-denied write, the post-dispatch uncertainty record, the overlay pre-commit assertion, the scratch-identity collision, the heldAtWrite stamp, and the provider-repair oracle.

The limitation this ticket cannot fix, named because an implementer must know it

A controller wired into the orchestrator cannot recover the orchestrator. ADR-0026 AC-8 already accepts this — "the actuator is orchestrator-resident; if the orchestrator dies there is no heal" — recorded so a future agent does not grant the actuator a second home without re-opening the privilege decision.

That limitation stopped being theoretical on 2026-08-09. The observed deployment was seen at 01:26 Z with its Memory Core unhealthy and its orchestrator healthy; ~38 minutes later both were unhealthy. So the plane reached the state where the consumer this ticket adds would itself be down — and the same structural argument that produced assertNotSelfLifecycleTarget's refusal ("the audit record dies with the writer") applies to the controller.

Consequence for scope, stated plainly: this ticket closes the gap for sibling services — Memory Core, Knowledge Base, the store, the model — which is the majority of the failure surface and worth shipping on its own. It does not close it for the orchestrator, and a reader should not infer otherwise from "the immune system now acts."

The orchestrator case needs a recovery principal outside the cohort, which is live divergence on Discussion #13871 (@neo-gpt-emmy's row D — out-of-cohort reconcile job, versus row B — resident privileged sidecar). This observation is evidence in that divergence: an in-cohort healer, sidecar or otherwise, shares a failure domain with what it heals, and a plane where both the orchestrator and Memory Core are simultaneously unhealthy is precisely the case that distinguishes the two rows.

Out of Scope

• Why Memory Core wedges — #16677 owns the mechanism, and this ticket must not become an implicit fix for it.
• The terminal-posture half (a service self-terminating when its own surface is unreachable) — #16677 AC-4 owns it; noted here as complementary.
• restart: policies — corrected: they already exist on the running overlays and on the observed external plane. They recover only exited containers, so they are neither a gap nor a fix for the alive-but-unhealthy case this ticket targets. Nothing to add here.
• Widening the closed action set, adding a privilege, or the phase-2 homeostatic controller (Discussion #13873).
• The data-integrity world, which already has its controller and is cited here only as the precedent.

Avoided Traps

• Restarting on a false unhealthy — and this is documented, not hypothetical. A deployment's own composition carries comments recording that these MCP healthchecks had reported unhealthy while the server was answering: a probe deadline shorter than the operation it waits on (a CPU-only inference canary, and an embed POST aborting at ~30 s on a five-minute cadence) tripped the runtime's 10 s ceiling. The remedy was tuning — interval: 60s, timeout: 45s, retries: 5, start_period: 90s.

  A controller that restarted on the first authoritative container-unhealthy would have repeatedly restarted a working server on that plane, and each restart would have destroyed the in-flight work whose slowness caused the red. #16646 is the in-repo precedent for the same class — a container probe's red could not distinguish could-not-run from did-not-answer.

  The mitigation is already available and should be used rather than invented: the runtime's retries × interval window is the debounce, so the controller keys on the container's unhealthy state, not on a probe sample — and ADR-0025's inherited false-positive-safe discipline ("a single advisory probe never triggers action; sustained + multi-fact only") is the governing rule, not a nicety. On a plane tuned as above that is five minutes of sustained failure before any action is even considered, which is the right order of magnitude and should be stated rather than assumed.

• "Nothing acts" as a blanket claim. It is false: DataIntegrityDiagnosisService:227 actuates today. Checking the other side of that boundary is what turned this from a vague immune-system complaint into a one-world wiring gap — and an earlier draft of this ticket carried the over-broad version.

• Reading selfHeal.total: 0 as "nothing has broken." It currently cannot distinguish that from "nothing consumes the decisions", which is why AC-5 requires recording the no-act decision.

• Restart-first as the mapping. An unhealthy-but-alive service may be mid-recovery; the cadence and threshold are the design content, and a controller that restarts on the first authoritative fact would be a new failure mode rather than a fix.

• Treating a container runtime restart policy as the answer. The observed process had not exited; restart: is exit-driven and never fires for it. This trap caught the author of this ticket in the opposite direction — an earlier revision asserted the policies were absent, from a grep of the base compose only, when they are declared in the overlay that actually runs. Both the absence claim and the it-would-fix-it claim are wrong, and for the same reason: the policy's trigger is process exit, so its presence or absence does not bear on this failure mode at all. Check which compose layer you grepped before asserting a deployment property.

Related

#16677 (the wedge mechanism and its AC-4 terminal posture) · #16636 (records this gap as an unowned design lane) · #16596 (same note) · #16706 (the deployment-readiness tracker this blocks — its stability definition is "it broke, we saw it, and it came back on its own", and this is the third clause) · ADR-0025 / ADR-0026 / ADR-0027 · Discussion #13871 (the origin ideation, whose §2.4 controller this implements) · Discussion #13873 (phase-2, out of scope)

Live latest-open sweep: checked latest 20 open issues created-descending at 2026-08-09T01:38Z; no equivalent found. A2A in-flight sweep: 30 most recent messages across all read-states; no [lane-claim]/[lane-intent] on this scope. Structure map run this session (npm run ai:structure-map -- --files --loc); the controller belongs beside its siblings in ai/daemons/orchestrator/services/, matching DataIntegrityDiagnosisService.

Origin Session ID: a641ddac-565a-4fc8-adc1-6c25629bddb7

Retrieval Hint: query_raw_memories("container-health diagnosis recorded never actuated reactive controller gap unhealthy alive") · ContainerHealthDiagnosisService.mjs:468 · DataIntegrityDiagnosisService.mjs:227