Restart churn is collected but never diagnosed

Narrowed 2026-08-03 after @neo-gpt-emmy's pre-implementation review, before any code. As filed, this conflated three different things — point-in-time liveness, restart stability, and independent control-plane observation — and proposed a fix that would not have worked. Both corrections are hers; the original framing is preserved in the first comment.

Context

Found while root-causing #16459. That ticket is why the orchestrator was dying. This one is why nobody found out.

RestartCount  977
State         running          (current incarnation ~28s old)
Health        healthy          (last 4 probes: exit=0)
inspect_deployment  →  healthy, no diagnostic facts

The container has died roughly 977 times. Every surface that could report it says it is fine.

The Problem

Restart churn is collected and never diagnosed.

DeploymentStateBridgeService.summarizeInspect() (:830-849) already records restartCount, startedAt, finishedAt, exitCode, oomKilled, state.status and state.health. git grep restartCount -- ai/ returns exactly one hit: that assignment. The number that would have made this obvious on day one is collected, published, and evaluated by nothing.

ContainerHealthDiagnosisService is the component that turns bounded read-observe facts into typed diagnoses, and its CONTAINER_HEALTH_FACT_TYPES covers container-down, container-unhealthy, resource-saturation, memory-saturation, and five more. Every one is point-in-time. None expresses a property across container incarnations, which is the only place this failure exists.

Why a "better healthcheck" is the wrong fix — and was the original proposal

A Docker healthcheck evaluates the current incarnation. During this loop each incarnation lived ~3 minutes and was genuinely alive; the process at the moment of writing was 28 seconds old and genuinely healthy. A process-bound probe would have passed too, on every one of the 977 incarnations, because at every instant the process really was running.

Restart churn is a history property. A point-in-time probe cannot express it no matter what it probes. The original version of this ticket proposed replacing the lease probe with something process-bound and asserted an AC that a crash-looping container should report unhealthy — that AC is unsatisfiable via healthcheck by construction. Correction is @neo-gpt-emmy's.

What must NOT change

• The authority-lease healthcheck stays exactly as it is. #16283 established it deliberately as the point-in-time event-loop liveness signal. It is not restart history and was never meant to be; using its freshness as a stability signal is the reader's error, not the probe's.
• The orchestrator-resident SPOF stays accepted. ADR-0025 AC-7 records it as an operator-accepted caveat, and states in terms: "recorded so a future agent does not 'fix' it by granting the actuator a second independent home without re-opening the privilege decision." This ticket does not re-open it. I wrote that AC and then filed the first version of this ticket proposing precisely what it warns against, without reading it.

What is genuinely stale in the original framing

The "nothing outside the cohort can observe this" claim is only half true. The graph-independent snapshot and the public KB/MC read path shipped in #13926 / #13914 — an external reader already exists. What does not exist is a continuously scheduled evaluator that consumes it and raises something. That is a different, larger piece of work, and it belongs in a design artifact rather than here.

The Architectural Reality

• ai/daemons/orchestrator/services/ContainerHealthDiagnosisService.mjs:8-32 — CONTAINER_HEALTH_FACT_TYPES, CONTAINER_HEALTH_ACTION_CLASSES, DEFAULT_CONTAINER_HEALTH_DIAGNOSIS_CONFIG. The extension point.
• ai/daemons/orchestrator/services/DeploymentStateBridgeService.mjs:830-849 — summarizeInspect(), the existing collection site; already carries every input needed.
• ADR-0025 AC-7 — the accepted orchestrator SPOF and its explicit do-not-fix-this-silently clause.
• ADR-0026 — the recovery actuator's privilege boundary.
• #16283 — the lease probe's point-in-time contract.
• #13926 / #13914 — the external read path that already exists.

The Fix

One typed fact, published through surfaces that already exist.

Extend ContainerHealthDiagnosisService with a restart-churn fact type, derived from data summarizeInspect() already collects, and publish it through the deployment-state bridge so inspect_deployment reports it.

The design questions that make this real rather than a counter with a number:

1. Threshold and window. Restarts-per-window, not a lifetime total; a container that restarted 30 times last month is not sick today.
2. Container-generation semantics. RestartCount resets when a container is recreated, so the baseline has to key on container identity — otherwise a redeploy silently clears the evidence, or a stale baseline reports churn that already ended.
3. Planned-restart exclusion. A deploy, a recreate, and a lifecycle restart all increment the count legitimately. If those trigger it, the fact is noise and will be ignored within a week.
4. Severity and action class. record, not restart. Restarting a crash-looping container is the one response guaranteed to make it worse, and CONTAINER_HEALTH_ACTION_CLASSES already separates the two.
5. The exact consumer. The deployment-state bridge, surfaced by inspect_deployment — the reader an operator or agent already calls.

Contract Ledger Matrix

Decision Record impact

aligned-with ADR-0025 / ADR-0026. It adds a fact type inside the accepted orchestrator-resident boundary and changes no privilege, no action class, and no lease contract. Any always-on external evaluator, notification path, or actuator is explicitly not this ticket — that re-opens ADR-0025 AC-7 and overlaps D#15758's unresolved authority work, and needs a design artifact that says so.

Acceptance Criteria

• Given a stable container generation whose restart count crosses the defined threshold within the defined window, existing deployment inspection reports a typed degraded diagnosis.
• Recreation and planned deployment restarts do not falsely trigger it.
• The churn baseline is keyed to container identity, so a recreate resets it rather than carrying a stale count.
• The threshold, window, and severity are configuration with stated defaults, not literals.
• The action class is record; the fact never actuates a restart.
• The fact is reachable through inspect_deployment without anyone running docker inspect.
• An unreadable inspect surfaces the existing runtime-read-failed path, never an implicit "no churn".
• The authority-lease healthcheck is unchanged, and #16283's point-in-time liveness contract still holds.
• No new privilege, socket grant, or lifecycle action; ADR-0025 AC-7 remains accepted and unmodified.

Out of Scope

• Changing the lease healthcheck. It is correct for what it measures.
• An always-on external evaluator, notifier, or actuator. Re-opens ADR-0025 AC-7 and overlaps D#15758. Separate design artifact.
• Why the orchestrator was dying. #16459; the ceilings are PR #16460.
• Proving those ceilings hold. #16463.
• The exit-code observation. ExitCode=0 alongside a fatal heap abort is odd and recorded on #16459 as an observation, not a claim.

Avoided Traps

• "Make the healthcheck process-bound." The original proposal. A point-in-time probe passes on every incarnation of a crash loop, because each incarnation really is alive — 977 times over.
• Reading lease freshness as a stability signal. The lease outliving process death is the property #16230 exists to provide. The defect was a reader expecting history from an instantaneous signal.
• Granting the observer a second home. ADR-0025 AC-7 anticipates this exact move and refuses it without a re-opened privilege decision.
• A raw counter with a threshold. Without generation semantics and planned-restart exclusion it fires on every deploy, and a signal that cries wolf is retired within a week — leaving the blind spot plus a disabled alarm.
• Assuming no external reader exists. #13926 / #13914 shipped one. The gap is a scheduled evaluator, not the read path.

Related

• #16459 — the OOM this concealed · #16460 — the ceilings · #16463 — proving they hold.
• #16283 — the lease probe's point-in-time contract, unchanged here.
• #13926 / #13914 — the existing external read path.
• ADR-0025 / ADR-0026 — the accepted SPOF and actuator privilege boundaries.

Live latest-open sweep: checked latest open issues at 2026-08-03T18:56Z; no equivalent found. Narrowed after pre-implementation review at 19:40Z; no code was written against the original shape.

Origin Session ID: 9f05cd72-5457-4ec2-926c-ef1406041f19

Retrieval Hint: query_raw_memories("restart churn collected never diagnosed typed fact ContainerHealthDiagnosisService point-in-time probe cannot express history")