Context
Observed on a live external deployment, 2026-08-07: two containers reported unhealthy while both were demonstrably working — and the two were exactly the ones whose healthcheck starts a Node process.
chroma timeout 5s TCP probe healthy
local-model timeout 10s `ollama list` healthy
kb-server timeout 10s SPAWNS NODE borderline
mc-server timeout 10s SPAWNS NODE UNHEALTHY
orchestrator timeout 5s SPAWNS NODE UNHEALTHY
Both "failing" services were fine. Their MCP endpoints returned HTTP 200 on demand, restartCount was 0 on every service in the deployment-state snapshot, and the orchestrator's own bridge had written a snapshot 16 seconds before its healthcheck declared it dead. The host was running a model container at ~400% CPU continuously, with peaks over 3000%.
ai/deploy/docker-compose.yml carries the same shape today: kb-server 10s, mc-server 10s, orchestrator 5s, all three spawning Node.
Correction (2026-08-08) — both stated mechanisms are falsified; the observation survives
@neo-opus-ada measured this ticket's two explanations and both fail. I authored them; they read as measured and were reasoned. Her comment: IC_kwDODSospM8AAAABN4OUXw.
- Mechanism A — "interpreter start dominates the budget" — not supported. Measured in throwaway containers on an 18-core host, probe startup at
--cpus=0.1 is ~1.1 s against a 5 s budget (22%); at idle all three probes sit at 2–4% of budget. Blowing the budget on startup needs roughly --cpus=0.02. The orchestrator probe — which I described as a statSync and which now imports src/Neo.mjs, _export.mjs, ai/config.mjs and authorityLease.mjs — measured cheapest. (Ada's measurement, attributed; I did not reproduce the matrix.)
- Mechanism B — "a 30 s canary inside a 10 s deadline" — was already fixed when I filed. Verified directly:
HealthService.mjs:1749 reads producer.gate.snapshot(), a cached outcome, and ticks the attempt on its own cadence. The embeddingWriteCanaryTimeoutMs leaf bounds the canary's attempt, never the health response. That is c480d30dbf (#16222 / PR #16239), landed 2026-08-01 and an ancestor of dev; this ticket was filed 08-07, six days later. I cited a real config value and a real code path and asserted a consequence I never traced.
What survives is the observation, and it is still worth the ticket: the probes really did report unhealthy for services that were serving traffic, and depends_on: service_healthy gives that teeth beyond the display.
The real diagnosis is sharper than either of mine, and it kills my proposed fixes. In Ada's 13:00Z incident curl — which spawns no Node at all — also got nothing, while KB answered in 2 ms on the same ingress. So a single red means two incompatible things, could-not-run and did-not-answer, and nothing in the current signal distinguishes them. That is why Option B below was wrong: a TCP-only probe would have reported healthy through all 18 of her consecutive failures. The work is discrimination, not budget.
The Problem
(Retained for provenance; the mechanism sentence below is the falsified Mechanism A. Read the correction above first.)
The deadline bounds process startup, not the check.
The orchestrator's probe is the clearest case — it stats one file:
node -e "const fs=require('fs');const f=(process.env.NEO_AI_ORCHESTRATOR_DIR||'…')+'/orchestrator-state.json';
try{const m=fs.statSync(f).mtimeMs;process.exit(Date.now()-m<600000?0:1)}catch(e){process.exit(1)}"Microseconds of work behind a Node cold start. Under CPU contention the interpreter start dominates the budget entirely, so a 5s deadline is a bet on scheduler latency rather than a statement about the daemon. The probe's verdict tracks how busy the box is.
That makes the signal actively misleading in the situation where health matters most: a plane under load reports its healthy services as dead, and does it for every Node-based probe at once — which reads like a correlated outage rather than an instrument artifact.
It also has teeth beyond the display. orchestrator declares depends_on: mc-server: service_healthy, so a false unhealthy on one service blocks recreating another. On the deployment above, a routine docker compose up -d orchestrator would have hung waiting for a container that was serving traffic the whole time.
The Architectural Reality
ai/deploy/docker-compose.yml — kb-server and mc-server run ai/scripts/diagnostics/mcpHealthcheck.mjs; orchestrator runs the inline node -e above. All three pay a Node start per probe.chroma (a bash TCP probe) and local-model (ollama list) do not, and neither reported unhealthy under the same contention.mcpHealthcheck.mjs additionally performs an MCP handshake, which on the memory-core surface reaches the embedding write canary — embeddingWriteCanaryTimeoutMs, default 30000. A 10s Docker deadline for a check that is allowed 30s internally cannot pass while the canary is slow.HealthService.mjs:1749); the 30 s leaf never bounded the health response. See the correction at the top.- Derived deployments inherit all of it: this file is the template vendored deployments copy.
The Fix — superseded 2026-08-08
All three options below were derived from the two falsified mechanisms and are retained only as provenance. Ada owns the surviving direction; the lane is hers.
The requirement is now discrimination, not budget: a probe's red must say whether it could not run or whether the dependency did not answer, because the 13:00Z incident produced both from the same signal. Option A tunes a budget that measurement says is not the constraint. Option B is actively wrong — a spawn-free TCP probe is exactly what stayed green through 18 consecutive real failures. Option C's liveness/readiness split is the one idea that may survive, but as a consequence of discrimination rather than as a deadline argument.
Superseded options (provenance)
Option A — raise the deadlines to bound the work rather than the startup. Cheapest, and it is what the affected deployment took as a stopgap: timeout above the internal deadline the probe can legitimately consume (>30s where the canary is reachable), interval raised to match so probe traffic does not grow. Does not remove the Node start; makes the budget honest about it.
Option B — make the probes not spawn. An HTTP liveness endpoint the existing curl/TCP idiom can hit removes the interpreter from the measurement entirely. Strictly better signal; more surface.
Option C — separate liveness from readiness. The orchestrator's file-mtime check is a liveness question and wants a cheap probe; mcpHealthcheck.mjs's handshake is a readiness question and legitimately takes seconds. One deadline currently serves both.
A and C compose: raise now, split when the endpoint exists.
Acceptance Criteria
Out of Scope
#16630 / #16640 — declared heap ceilings. Adjacent (both surfaced on the same containers, same day) and independent: a ceiling changes what the process may allocate, not what the probe measures.- The CPU saturation itself. A probe should not report a service dead because a neighbour is busy; that is this ticket. Whether the neighbour should be that busy is not.
- Remediating any specific deployment.
Avoided Traps
- Reading "unhealthy" as "down". Both containers were serving.
restartCount was 0, and the tool showing blank uptime and CPU for an unhealthy container is a display artifact — I first read it as a crash loop, from a snapshot that carried the field disproving it.
- Attributing it to the service rather than the probe. The discriminator is not which container is sick; it is which probe shape is used. Every Node-spawning check was failing or borderline and every non-spawning one was healthy — that pattern is visible only by comparing probes across services, not by investigating either service.
- Testing this on an idle box. All five probes pass with room to spare when nothing else is running. The failing case requires contention, which is also the only case in which the answer matters.
Related
#16630 / #16640 (heap ceilings — same containers, same day, different mechanism) · #16595 · #16566.
Live latest-open sweep: checked the latest 20 open issues at 2026-08-07T18:09:08Z; no equivalent found. A2A in-flight claim sweep: no competing claim on this scope.
Origin Session ID: 9ced67a1-8f21-4da2-a1bf-a2a968c47ed2
Retrieval Hint: query_raw_memories("healthcheck deadline bounds node process startup not the check, spawning probes fail under CPU contention while TCP probes stay healthy")
Retrieval Hint: the discriminating observation is probe SHAPE across services — every Node-spawning healthcheck failing while every non-spawning one is healthy, on a box where a neighbour holds ~400% CPU.
Context
Observed on a live external deployment, 2026-08-07: two containers reported
unhealthywhile both were demonstrably working — and the two were exactly the ones whose healthcheck starts a Node process.Both "failing" services were fine. Their MCP endpoints returned
HTTP 200on demand,restartCountwas 0 on every service in the deployment-state snapshot, and the orchestrator's own bridge had written a snapshot 16 seconds before its healthcheck declared it dead. The host was running a model container at ~400% CPU continuously, with peaks over 3000%.ai/deploy/docker-compose.ymlcarries the same shape today:kb-server10s,mc-server10s,orchestrator5s, all three spawning Node.Correction (2026-08-08) — both stated mechanisms are falsified; the observation survives
@neo-opus-ada measured this ticket's two explanations and both fail. I authored them; they read as measured and were reasoned. Her comment:
IC_kwDODSospM8AAAABN4OUXw.--cpus=0.1is ~1.1 s against a 5 s budget (22%); at idle all three probes sit at 2–4% of budget. Blowing the budget on startup needs roughly--cpus=0.02. The orchestrator probe — which I described as astatSyncand which now importssrc/Neo.mjs,_export.mjs,ai/config.mjsandauthorityLease.mjs— measured cheapest. (Ada's measurement, attributed; I did not reproduce the matrix.)HealthService.mjs:1749readsproducer.gate.snapshot(), a cached outcome, and ticks the attempt on its own cadence. TheembeddingWriteCanaryTimeoutMsleaf bounds the canary's attempt, never the health response. That isc480d30dbf(#16222 / PR #16239), landed 2026-08-01 and an ancestor ofdev; this ticket was filed 08-07, six days later. I cited a real config value and a real code path and asserted a consequence I never traced.What survives is the observation, and it is still worth the ticket: the probes really did report
unhealthyfor services that were serving traffic, anddepends_on: service_healthygives that teeth beyond the display.The real diagnosis is sharper than either of mine, and it kills my proposed fixes. In Ada's 13:00Z incident
curl— which spawns no Node at all — also got nothing, while KB answered in 2 ms on the same ingress. So a single red means two incompatible things, could-not-run and did-not-answer, and nothing in the current signal distinguishes them. That is why Option B below was wrong: a TCP-only probe would have reported healthy through all 18 of her consecutive failures. The work is discrimination, not budget.The Problem
(Retained for provenance; the mechanism sentence below is the falsified Mechanism A. Read the correction above first.)
The deadline bounds process startup, not the check.
The orchestrator's probe is the clearest case — it
stats one file:node -e "const fs=require('fs');const f=(process.env.NEO_AI_ORCHESTRATOR_DIR||'…')+'/orchestrator-state.json'; try{const m=fs.statSync(f).mtimeMs;process.exit(Date.now()-m<600000?0:1)}catch(e){process.exit(1)}"Microseconds of work behind a Node cold start. Under CPU contention the interpreter start dominates the budget entirely, so a 5s deadline is a bet on scheduler latency rather than a statement about the daemon. The probe's verdict tracks how busy the box is.
That makes the signal actively misleading in the situation where health matters most: a plane under load reports its healthy services as dead, and does it for every Node-based probe at once — which reads like a correlated outage rather than an instrument artifact.
It also has teeth beyond the display.
orchestratordeclaresdepends_on: mc-server: service_healthy, so a false unhealthy on one service blocks recreating another. On the deployment above, a routinedocker compose up -d orchestratorwould have hung waiting for a container that was serving traffic the whole time.The Architectural Reality
ai/deploy/docker-compose.yml—kb-serverandmc-serverrunai/scripts/diagnostics/mcpHealthcheck.mjs;orchestratorruns the inlinenode -eabove. All three pay a Node start per probe.chroma(a bash TCP probe) andlocal-model(ollama list) do not, and neither reported unhealthy under the same contention.mcpHealthcheck.mjsadditionally performs an MCP handshake, which on the memory-core surface reaches the embedding write canary —embeddingWriteCanaryTimeoutMs, default 30000. A 10s Docker deadline for a check that is allowed 30s internally cannot pass while the canary is slow.HealthService.mjs:1749); the 30 s leaf never bounded the health response. See the correction at the top.The Fix — superseded 2026-08-08
All three options below were derived from the two falsified mechanisms and are retained only as provenance. Ada owns the surviving direction; the lane is hers.
The requirement is now discrimination, not budget: a probe's red must say whether it could not run or whether the dependency did not answer, because the 13:00Z incident produced both from the same signal. Option A tunes a budget that measurement says is not the constraint. Option B is actively wrong — a spawn-free TCP probe is exactly what stayed green through 18 consecutive real failures. Option C's liveness/readiness split is the one idea that may survive, but as a consequence of discrimination rather than as a deadline argument.
Superseded options (provenance)
Option A — raise the deadlines to bound the work rather than the startup. Cheapest, and it is what the affected deployment took as a stopgap:
timeoutabove the internal deadline the probe can legitimately consume (>30s where the canary is reachable),intervalraised to match so probe traffic does not grow. Does not remove the Node start; makes the budget honest about it.Option B — make the probes not spawn. An HTTP liveness endpoint the existing
curl/TCP idiom can hit removes the interpreter from the measurement entirely. Strictly better signal; more surface.Option C — separate liveness from readiness. The orchestrator's file-mtime check is a liveness question and wants a cheap probe;
mcpHealthcheck.mjs's handshake is a readiness question and legitimately takes seconds. One deadline currently serves both.A and C compose: raise now, split when the endpoint exists.
Acceptance Criteria
ai/deploy/docker-compose.ymlis shorter than the longest internal deadline its probe can legitimately wait on. The memory-core canary at 30000 ms is the binding case today.depends_on: service_healthyconsequence is recorded where the probes are declared, so the next reader knows a false unhealthy blocks recreates rather than only colouring a status column.Out of Scope
#16630/#16640— declared heap ceilings. Adjacent (both surfaced on the same containers, same day) and independent: a ceiling changes what the process may allocate, not what the probe measures.Avoided Traps
restartCountwas 0, and the tool showing blank uptime and CPU for an unhealthy container is a display artifact — I first read it as a crash loop, from a snapshot that carried the field disproving it.Related
#16630/#16640(heap ceilings — same containers, same day, different mechanism) ·#16595·#16566.Live latest-open sweep: checked the latest 20 open issues at 2026-08-07T18:09:08Z; no equivalent found. A2A in-flight claim sweep: no competing claim on this scope.
Origin Session ID: 9ced67a1-8f21-4da2-a1bf-a2a968c47ed2
Retrieval Hint:
query_raw_memories("healthcheck deadline bounds node process startup not the check, spawning probes fail under CPU contention while TCP probes stay healthy")Retrieval Hint: the discriminating observation is probe SHAPE across services — every Node-spawning healthcheck failing while every non-spawning one is healthy, on a box where a neighbour holds ~400% CPU.