A degraded Memory Core gates ingress and orchestrator startup

Context

Observed 2026-08-03 on the one-machine local Docker Agent OS, during the first cold boot after a host shutdown. Memory Core reported degraded for ~12 minutes because its embedding write canary could not reach the provider (LM Studio had not been started). Measured live during that window:

mc-server      state=running  health=unhealthy   (FailingStreak 56)
ingress        state=running  health=healthy
orchestrator   state=running  health=healthy
kb-server      state=running  health=healthy

MC was serving correctly the entire time — healthcheck answered over the ingress, the WAL measured caught-up (depth 0), and every startup dependency was ready. The only fault was one provider-dependent canary.

The Problem

Memory Core deliberately treats non-WAL dependencies as best-effort. From Server.mjs:427-430:

"Graph, wake-subscription, inference, and vector/session readiness must be observable but must not veto MCP server startup; otherwise the mandatory add_memory final-turn save disappears exactly when a degraded deployment most needs lossless WAL capture."

The container healthcheck contradicts that contract. mcpHealthcheck.mjs defaults --expected-status healthy, so the deliberately-survivable degraded exits non-zero and the container is marked unhealthy. Both ingress and orchestrator then declare depends_on: mc-server: condition: service_healthy.

This boot escaped the consequence only because depends_on conditions gate docker compose up, not restart-policy restarts — the containers already existed and came back independently. The table above is the direct evidence: ingress reached healthy while mc-server never did.

The documented start command in ai/scripts/lifecycle/local-agent-os/README.md is:

docker compose --env-file .env -f ai/deploy/docker-compose.yml \
  -f ai/deploy/docker-compose.local-agent-os.yml \
  --profile cloud --profile ingress up -d --wait

On a cold host whose embedding provider is not yet up, that command waits on mc-server reaching service_healthy — which it cannot, because the missing provider is precisely what keeps it degraded. ingress never starts, 127.0.0.1:3102 never opens, and the Knowledge Base becomes unreachable too, despite kb-server being perfectly healthy. A provider-side degradation takes down the whole loopback plane, including the half that does not depend on the provider.

The Architectural Reality

• ai/deploy/docker-compose.yml:208-213 — mc-server healthcheck runs mcpHealthcheck.mjs with no --expected-status; interval 10s, retries 12, start_period 45s (≈165s of degraded → unhealthy).
• ai/deploy/docker-compose.yml:320-326 — orchestrator depends_on chroma + kb-server + mc-server, all condition: service_healthy.
• ai/deploy/docker-compose.local-agent-os.yml:82-86 — ingress depends_on kb-server + mc-server, condition: service_healthy.
• ai/scripts/diagnostics/mcpHealthcheck.mjs — --expected-status defaults to healthy; exits non-zero on anything else.
• ai/mcp/server/memory-core/Server.mjs:424-453 — prepareStartupDependency(), best-effort by design.
• ai/mcp/server/memory-core/toolService.mjs:197-219 — composeMemoryCoreHealthcheck().

The mismatch is a contract disagreement between two layers: the MCP server says "degraded is survivable, keep serving", and the container gate says "degraded is dead, block everything downstream".

The Fix

Decide what degraded means to Docker, then make both layers agree. Candidate shape, open to challenge:

• mc-server's container healthcheck treats degraded as alive (an --expected-status that accepts a set, or a dedicated liveness expectation), leaving unhealthy as the only failing verdict; and/or
• ingress / orchestrator gate on service_started for mc-server, so a degraded-but-serving Memory Core cannot black-hole the ingress.

The invariant to preserve: a provider-side degradation must not remove the plane's ingress, and must never take the Knowledge Base down with it.

Contract Ledger Matrix

Decision Record impact

none — deployment-topology correction; no ADR authority is amended or challenged.

Acceptance Criteria

• [L3-deferred — operator handoff needed] With the embedding provider unreachable from a cold start, the runbook's up -d --wait command brings ingress up and 127.0.0.1:3102/kb/mcp answers healthy. Fails RED today.
• [L3-deferred — operator handoff needed] A degraded Memory Core no longer gates any other service's startup; witnessed from a live run, not reasoned.
• A genuinely unhealthy Memory Core still fails its container healthcheck — the change must not flatten unhealthy into alive. (unit bound green; the live counterpart rides the same cold-provider handoff.)
• [L3-deferred — operator handoff needed] The runbook's start command is re-verified end-to-end against a cold provider.

Out of Scope

• The cold-boot ordering gap itself (Colima has no auto-start; LM Studio's server and model preload are manual). Separate concern.
• The canary's backoff behavior — measured correct: it self-cleared without intervention once the provider answered, exactly as configBase.mjs:413-419 documents.
• Who owns starting the provider.

Avoided Traps

• Do not relax the healthcheck to always pass. That destroys the signal unhealthy carries; the goal is to distinguish two verdicts, not erase both.
• Do not conclude "the reboot recovered, so there is no bug." The reboot path bypasses depends_on entirely. The path that breaks is the documented one a fresh contributor runs.
• Do not fix this by making the canary non-degrading. The canary is correct — it reports a real provider fault. The defect is what Docker does with that report.

Related

• #14124 / #12972 — the embed-canary catch-22 at the MCP tool-gate layer (read-only diagnostics and add_memory blocked by the same canary). This ticket is the same tension one layer down, at the container/compose gate; the resolution there (exempt what must stay available) is the precedent.
• #16206 — ai/deploy composable layering.

Origin Session ID: 6fbb7047-4b3f-4842-af7d-0aa5949dc392

Retrieval Hint: "mc-server degraded unhealthy depends_on service_healthy ingress blocked cold boot" Retrieval Hint: "embedding provider absence gates loopback ingress 3102 KB unreachable"

Live latest-open sweep: checked the latest 20 open issues at 2026-08-03T08:43:35Z; A2A in-flight claim sweep over 30 messages (all read-states) at the same time. No equivalent found.