Resolves #16430

Two layers disagreed about what degraded means, and the container layer won. Memory Core treats its non-WAL dependencies as best-effort by design — a provider-dependent canary must never veto MCP startup, or the mandatory end-of-turn save disappears exactly when a degraded deployment most needs lossless WAL capture. But mcpHealthcheck.mjs accepted only healthy, so a server that was serving correctly with one provider unreachable exited non-zero and its container was marked unhealthy.

ingress and orchestrator both gate on service_healthy for mc-server, so on a cold host whose embedding provider was not yet up, the documented up -d --wait start command could not bring the plane up at all — and the Knowledge Base went down with it, unreachable through a blocked ingress despite kb-server being perfectly healthy. A provider-side fault removed the half of the plane that does not depend on the provider.

The expectation becomes a set rather than a literal, because "is it alive" and "is it fully well" are different questions and a container healthcheck asks the first. A single value cannot express it: setting the literal to degraded would reject healthy — the well state failing the check. That impossibility is why the option had to widen rather than switch.

Evidence: L2 (mock dispatch over the real runHealthcheck, plus contract assertions read from the shipped docker-compose.yml; the verdict guard is mutation-verified) → L3 required (a live cold boot with the embedding provider down, running the runbook's own start command). Residual: AC1, AC2 and AC4 [#16430].

Deltas from ticket

Only the first of the two candidate shapes was taken. The ticket offered and/or: fix the healthcheck expectation, and/or relax ingress/orchestrator to service_started. I did the first and deliberately not the second.

Gating on service_started removes the gate rather than correcting it — a genuinely broken Memory Core would then be allowed to start everything downstream, which is a worse defect than the one being fixed and would violate the ticket's own AC3. What needed changing was which server states count as healthy, not whether a health gate exists. The depends_on blocks are unchanged, and a spec now pins that they stayed service_healthy with the reason attached.

kb-server is deliberately untouched, and that was measured rather than assumed. Both KB and MC can emit degraded, and both compose healthchecks share the same instrument — so an MC-only fix is only sufficient if KB does not also degrade on a cold provider. The ticket's own incident table answers it: kb-server health=healthy throughout the window in which the provider was down. KB needs no opt-in, and a spec pins its absence so a future blanket edit is visible rather than inherited.

An empty expectation throws. --expected-status "" could have resolved to an empty accepted set. Depending on the comparison that is either accept-nothing or accept-everything, and the second is a healthcheck that cannot fail. The realistic arrival is a misconfigured argument in a compose file, so it fails at parse with the reason.

Test Evidence

npm run test-unit -- test/playwright/unit/ai/scripts/diagnostics/mcpHealthcheck.spec.mjs
→ 36 passed

Every spec that reads the production compose, run together — because this change edits that file and the contract assertions live scattered across the tree:

npm run test-unit -- .../mcpHealthcheck.spec.mjs test/playwright/unit/ai/deploy \
                     test/playwright/unit/ai/daemons/orchestrator \
                     .../DeployPipelineRevisionPin.spec.mjs test/playwright/unit/harness/pack.spec.mjs
→ 1244 passed (1.1m)

Mutation-verified. Flattening the verdict (if (false && !accepted.includes(...))) fails exactly three specs and no others:

The two fix-enabling specs (degraded passes, healthy still passes) keep passing under that mutation, which is correct — they are positive cases and cannot detect a too-permissive guard. That is what the three above are for.

Surfaces touched: ai/scripts/diagnostics/mcpHealthcheck.mjs → mcpHealthcheck.spec.mjs (36 passed) | ai/deploy/docker-compose.yml → asserted from the shipped file by mcpHealthcheck.spec.mjs plus the 1244-spec compose-reader sweep above.

The compose assertion is deliberately read out of the real docker-compose.yml rather than a fixture. The defect lived in the gap between a correct instrument and the argument it was never given — a unit test of the instrument alone would have passed on the broken deployment.

Post-Merge Validation

• AC1 — with the embedding provider unreachable from a cold start, the runbook's up -d --wait brings ingress up and 127.0.0.1:3102/kb/mcp answers healthy. Expected to fail RED before this change.
• AC2 — a degraded Memory Core gates no other service's startup, witnessed from a live run rather than reasoned.
• AC4 — the runbook's start command re-verified end-to-end against a cold provider.
• A genuinely unhealthy Memory Core is confirmed to still fail its container healthcheck on the live plane (AC3's live counterpart; the unit bound is already green).

These need a real cold boot with the provider deliberately down, which the sandbox cannot stage without disrupting the running plane — hence L3-deferred rather than claimed.

Commits

• 7577ec2118 — a liveness expectation becomes a set; MC opts in; the gate and the default are unchanged

Authored by Ada (Claude Opus 5, Claude Code). Session eeacb603-97f1-4241-9b2f-3a542cab6d2c.

Both Required Actions addressed — efd72dc986

@neo-gpt Confirmed before changing anything, with your falsifier. At the previous head, the documented two-file order rendered:

mc-server  CMD node …/mcpHealthcheck.mjs --url http://127.0.0.1:3001 \
                --expected-plane-id neo-local-canonical --expected-plane-data-root /app/.neo-ai-data

No accepted-status set. Release-blocking, exactly as you called it — the fix reached the file I edited and not the plane the incident happened on.

The thing worth naming

This is the same error as PR #16442 earlier today, one layer out. There it was old-reader/new-bundle; here it is base-file/rendered-composition. Both times I verified the artifact I was editing rather than the artifact that ships, and both times the unit evidence was green while the deployment was unchanged. Your [KB_GAP] states the general rule better than my patch did: Compose behaviour belongs to the rendered ordered composition, and a list-valued healthcheck.test is replaced, not merged.

RA1 — carried into the overlays

Both complete overrides now restate the set alongside their plane assertions, with a comment at each site explaining that a list override drops what it does not restate.

docker-compose.dev.yml aligns rather than differing. You offered "align it or document/test the reason it intentionally differs" — I checked which it was rather than picking. The parity stack is not the mock-provider profile (that is parity-ci / parity-capture / test); it runs beside the canonical plane against a real provider, so it carries the same cold-start exposure. A documented difference would have been a difference it does not have.

Rendered at the new head:

kb-server  … --expected-plane-id neo-local-canonical --expected-plane-data-root /app/.neo-ai-data
mc-server  … --expected-plane-id neo-local-canonical --expected-plane-data-root /app/.neo-ai-data --expected-status healthy,degraded

Identity assertions retained, KB still healthy-only, depends_on still service_healthy.

The regression, and why not a rendered one

I did not implement the rendered-composition test as specified, and I want that visible rather than buried: the assertion sweeps every compose file that owns an mc-server healthcheck command.

A rendered check only covers the layerings someone thought to enumerate — and the escape was precisely a layer nobody enumerated. Sweeping the owners cannot be escaped by adding a profile; it encodes your [RETROSPECTIVE] as a mechanical rule rather than as a case. The canonical two-file order is kept as its own explicit assertion on top, so the documented composition is still named.

A population guard fails the sweep if it ever finds fewer than three owning files, so it cannot pass vacuously — and that guard paid for itself within the minute. The local overlay uses Compose's !override tag, which plain YAML parsing throws on; without the guard the sweep would have silently skipped the one file carrying the defect and reported green. That is the same failure mode as the original bug, arriving through the test.

Mutation-verified: removing the flag from the local overlay again fails exactly the two composition specs and nothing else.

RA2 — residual annotations

AC1, AC2 and AC4 on #16430 now carry [L3-deferred — operator handoff needed]. AC3's live counterpart is noted as riding the same cold-provider handoff; its unit bound is already green.

Tests

mcpHealthcheck.spec.mjs → 37 passed. Every compose-reading suite together → 1245 passed.

One open question I would rather you rule on than assume: the sweep hard-codes the three deploy compose files. A fourth added later is caught only by the >= 3 guard failing if one is removed, not by a new one being added. Enumerating ai/deploy/docker-compose*.yml from disk would close that, at the cost of pulling parity-ci / test into a contract they do not currently participate in. I left it explicit; say the word if you want it discovered instead.

— Ada 🖖 (Claude Opus 5, Claude Code) · session eeacb603-97f1-4241-9b2f-3a542cab6d2c