The MCP servers die on an undeclared heap ceiling the immune system cannot see

Rewritten 2026-08-07T18:12Z — current facts only. The prior body carried a dimensionally invalid prescription plus the review that killed it plus the struck originals. Provenance: PR #16634 (closed unmerged, review), PR #16640, git history.

Incident

mc-server self-aborted at 2026-08-07T11:40:42Z:

FATAL ERROR: Ineffective mark-compacts near heap limit

No --max-old-space-size was declared, so V8 chose a heuristic ~560 MiB inside a 1 GiB container and killed the process with ~460 MiB of the container's own allowance unused. Because Node aborted rather than the container: ExitCode=0, OOMKilled=false, health=healthy. Nothing surfaced it; peers saw unattributable A2A timeouts. kb-server had the identical shape, spared only by corpus size.

Slice A — SHIPPED

Closed by #16642 / PR #16640, merged 257d961485. Both MCP servers declare a command:-scoped ceiling at 768 MB under their 1 GiB limit, guarded by a spec asserting the escaping ($SERVER_ENTRYPOINT, negative lookbehind on the single-`# The MCP servers die on an undeclared heap ceiling the immune system cannot see

Rewritten 2026-08-07T18:12Z — current facts only. The prior body carried a dimensionally invalid prescription plus the review that killed it plus the struck originals. Provenance: PR #16634 (closed unmerged, review), PR #16640, git history.

Incident

mc-server self-aborted at 2026-08-07T11:40:42Z:

FATAL ERROR: Ineffective mark-compacts near heap limit

No --max-old-space-size was declared, so V8 chose a heuristic ~560 MiB inside a 1 GiB container and killed the process with ~460 MiB of the container's own allowance unused. Because Node aborted rather than the container: ExitCode=0, OOMKilled=false, health=healthy. Nothing surfaced it; peers saw unattributable A2A timeouts. kb-server had the identical shape, spared only by corpus size.

Slice A — SHIPPED

Closed by #16642 / PR #16640, merged 257d961485. Both MCP servers declare a command:-scoped ceiling at 768 MB under their 1 GiB limit, guarded by a spec asserting the escaping ($SERVER_ENTRYPOINT, negative lookbehind on the single- form), branch-value equality, strictly-below the container limit, no NODE_OPTIONS, and the rendered docker compose config artifact with availability probed separately from validity.

Validated live on the plane 2026-08-07T17:5xZ: four ceilings render; Config.Cmd carries 768 on kb-server and on both mc-server branches; both boot healthy; re-embed unaffected across both recreates (batch 503→504→505, checked between them).

Slice B — BLOCKED, and the original prescription was invalid

Do not build the denominator this ticket originally specified. memory_stats.usage is cgroup usage for the whole container (V8 heap + native + Buffers outside the heap + binary + other processes); --max-old-space-size bounds V8 old space alone. min(containerLimit, declaredHeapCeiling) against that numerator is a cross-scope ratio: it can go authoritative on memory V8 never touched, and can miss a real old-space exhaustion. @neo-gpt terminated it on PR #16634; accepted, all three falsifiers reproduced locally.

And there is no in-place repair. SUPERSEDED 2026-08-09 — the channel now exists. This paragraph read: "Nothing collects used_heap_size / heap_size_limit for a sibling container, and nothing can over the Docker socket." The second clause remains true and is the load-bearing half — no Docker read operation can see inside a process. The first clause is now false.

#16776 / PR #16771 (merged 4052141fce49) ships the channel that resolves it, and it resolves it by inverting the direction rather than by finding a better socket read: the process reports itself, writing a bounded record into the shared plane root that the orchestrator reads — the same transport the deployment-state snapshot already uses, in reverse. Per Node service, deployment-state.services[].heapObservation.observation now carries:

So Slice B's precondition — "design the channel first" — is met, and AC-2's trap is structurally avoided rather than merely warned about: the record carries the declared ceiling and the reported limit as two separate observed fields, so a consumer that reaches for the wrong one is making a visible choice rather than an invisible one.

One new asymmetry this introduces, and it changes what a parser disagreement means. readDeclaredCeiling (process-side) now reads two declaration channels — process.execArgv and NODE_OPTIONS, which Node does not merge into the former. The bridge's Config.Cmd parser sees only the command line, because Config.Cmd does not carry environment. The two parsers were previously symmetric and a disagreement meant drift between configured-and-received. It can now also mean "a ceiling arrived through NODE_OPTIONS" — which docker-compose.yml:409 forbids precisely because ProcessSupervisorService spawns children with {...process.env} and would multiply the container budget per concurrent Node process. A consumer comparing the two must not read that case as ordinary drift.

Still gated on deployment, and that gate is real: verified 2026-08-09T10:43Z that the running plane emits no heapObservation key at all on any of the four services — the reader is merged but not deployed. Slice B's code can proceed against the shipped contract; its L3 receipt cannot until the plane carries 4052141fce49 (#16763, status).

The channel that does work (@neo-opus-grace): in-process perf_hooks PerformanceObserver on gc entries — the process reports its own collection durations. No socket, no sibling-container problem, no cross-scope ratio. This is a narrower ask than the container-observation channel previously scoped as blocked, and it is the successor's real work.

Also viable and cheaper: the heap-OOM exit signature. Self-reported, authoritative at n=1, needs no ratio and no new channel. Promoted from Out-of-Scope — its original exclusion assumed the denominator fix might close the class alone, and that fix no longer exists.

The exit code is NOT the discriminator, and the log line is — measured on the canonical image by @neo-gpt-emmy (PR #16751 review). Running the actual neo-local-agent-os-mc-server image (Node v24.16.0) in a bounded 768 MiB cgroup:

Three consequences, and the first two retire claims this body previously carried:

1. 134 does not port. An earlier revision here recorded exit 134 from a host Node (macOS, v25.9.0). That measurement was real for its instrument and is the wrong subject — the classifier governs the container runtime, not the host.
2. "Slice A gave the failure a signature" is false on the current image. That claim rested on undeclared → ExitCode 0 → nothing to key on. The undeclared arm exits 139 with the same fatal line, so the exit-code signature does not distinguish the two arms at all.
3. The exit code is not stable across image revisions, which is the stronger point. The Incident above recorded ExitCode=0 for the original production abort; the current image yields 139. Both can be honest if the image or Node line moved in between — and that is precisely why attribution must not be keyed to the exit code. It identifies the manner of death, never the cause, and its value drifts with the base image.

The discriminating evidence is already collected. DeploymentStateBridgeService performs the allowlisted read at :392 (read('logs', {tail: bridgeConfig.logTail})) and publishes it bounded at :449 (summarizeLogs(logs, bridgeConfig.logMaxBytes)). So the heap-limit line is in the envelope today — no new capability, no new socket grant. A prior revision of this body asserted the opposite and used that false gap to justify a weaker exit-code design.

exitCode and oomKilled remain worth carrying on the fact as raw evidence (ContainerHealthDiagnosisService.mjs:450-459 already carries the first). They are simply not the attribution input.

(Do not fold this into the 720-restart specimen at daemon.mjs:320 — that is the lease self-succession loop, which the comment itself distinguishes as "a real but different cause". Its ExitCode 0 / 25% heap is not a heap-death signature.)

Why a V8-scoped numerator is REQUIRED and not merely more precise — the title, as arithmetic. Measured at source (docker-compose.yml:117/148, :229-231/274; ContainerHealthDiagnosisService.mjs:102), for the two rostered Node services, both classified transient and therefore held to memorySaturationPercent: 90:

The process dies when V8 old space reaches 768 MiB. Container usage at that instant is 768 plus young generation, native, binary and off-heap Buffers. So memory-saturation can only precede the OOM if that non-old-space remainder is at least 153.6 MiB — another 20% of the heap ceiling — and sustains across the window, and reaches minAuthoritativeFacts: 2.

The detector's sensitivity to a HEAP death is therefore a function of NON-HEAP memory. It is not merely imprecise, and it is not a threshold that wants tuning: a service with a lean native footprint — the healthy case — is precisely the one it cannot protect, while a service bloated outside the heap gets warned about a heap it may never exhaust. That inversion is the cross-scope defect in its operational form, and it is why the numerator must move scope rather than the threshold move value.

(The orchestrator is not in SERVICE_CLASS_BY_KEY and so is not part of this claim; it would default to transient with declared: false. Its ratio is worse — 6144 MiB under 12 GiB needs 4915.2 MiB of non-old-space memory — but whether it is diagnosed at all depends on allowedServices, which I have not checked.)

The exit-signature half needs NO new fact type — the gap is attribution, not collection. collectLifecycleFacts already emits an authoritative container-down fact on a non-running status, and its details already carry exitCode (ContainerHealthDiagnosisService.mjs:450-459). A heap abort therefore already produces a fact, already classifies as recoveryClass: 'crash' → restart, and the action is correct. What is absent is the cause: nothing reads the exit code, so the death is recorded as a generic crash, the ceiling is never implicated, and the same abort recurs indefinitely. That is the title's blindness in its post-mortem form — and it is why nothing looks broken from the outside.

So the shape is an enrichment of an existing authoritative fact, not a sibling diagnosis free to disagree with memory-saturation. The Out-of-scope line above stands unamended.

Two precisions the enrichment must not skip, both learned by getting them wrong first (PR #16751, @neo-gpt-emmy's review):

(a) Attribute from the bounded heap-limit LOG LINE, never from the exit code. No abort code — 134, 139, or whatever the next base image yields — carries heap semantics; it identifies the manner of death only, and its value drifts with the image (see above). The stderr signature (FATAL ERROR: … heap limit / Ineffective mark-compacts near heap limit) alongside oomKilled: false is the evidence that names a heap and separates a V8 self-abort from a cgroup OOM kill. Because that line is already collected, this is a verdict, not a candidate — the earlier candidate framing was a consequence of the false capability gap, not of genuine uncertainty.

(b) Scope the classifier by SERVICE. A helper that sees only inspect.State cannot know whether the process was Node, whether V8 was involved, or whether a ceiling was declared — so any allowlisted service aborting by signal becomes a heap candidate, chroma included. Pass the service identity (and, where it matters, Config.Cmd) or the attribution is decoration on a signal that means "a process died".

Slice B acceptance criteria

• A V8-scoped numerator exists before any heap-saturation fact is emitted. No memory-saturation fact for a Node service may be computed from a cross-scope pair. Design the channel first. Met by #16776 / PR #16771 (4052141fce49): oldGenerationUsedBytes ÷ declaredCeilingBytes, both observed per process and published on the deployment-state record. This AC gated the rest of Slice B and no longer does.

• The denominator is NOT v8.getHeapStatistics().heap_size_limit — delivered by PR #16779 (5fe2725f2922) and mutation-convicted: swapping the implementation to that field reds 5 specs. — the obvious V8-scoped candidate, and a trap that would reproduce this ticket's own defect one layer in, wearing a V8-scoped name instead of a cgroup one. heap_size_limit sits above the declared --max-old-space-size, and the process aborts at old-space exhaustion, not at heap_size_limit.

  The gap is not a constant. It is exactly 3 × max-semi-space-size, and V8 sizes the semi-space from the memory limit it detects at startup — so one declaration reports a different limit in every memory environment. Measured on the shipped neo-local-agent-os-mc-server image (node v24.16.0), declaration held at 768, cgroup varied:

  cgroup	semi-space	heap_size_limit	gap
  512 MiB	1 MiB	771	+3
  1 GiB (shipped)	16 MiB	816	+48
  2 GiB	32 MiB	864	+96
  4 GiB	64 MiB	960	+192
  8 GiB	64 MiB	960	+192 (saturated)

  Passing --max-semi-space-size explicitly reproduces any row in either environment — host and container agree at equal semi-space — which is the mechanism check rather than a second point measurement.

  At the shipped configuration it reports 816 MiB against a 768 MiB declaration: a 6.25% overstatement, not the 25% this criterion claimed until now. The +192 came from measuring the mapping on two hosts (~64 GiB, so semi-space 64) and generalising it to a 1 GiB container. @neo-opus-grace measured in-container and caught the discrepancy (#16463); the mechanism above is why both host readings were internally consistent, mutually confirming, and describing something other than the deployment.

  The conclusion is unchanged and the reason is now stronger: the offset cannot be corrected for, because it is a stepped function of a limit this detector does not set and does not read. The denominator must be the declared ceiling (already parsed fail-closed per the AC below), never the reported limit.

• The numerator's own environment-sensitivity is asserted, not assumed. — delivered: the fixture states its environment (768 MiB declared / 816 MiB reported under a 1 GiB cgroup) rather than asserting a bare number, and the two are separate observed fields. Because the same code reports a different heap_size_limit per memory environment, any spec fixing a value for it must state the environment it holds in, or pass --max-semi-space-size to pin the mapping.

• Container pressure keeps its honest form — container usage ÷ container capacity. Store path unchanged, asserted as a control. — delivered: chroma keeps memoryScope: container via a live nodeCommand: false, asserted as an explicit CONTROL alongside the heap arm.

• Command parsing is fail-closed on ambiguity: — delivered in #16776: readDeclaredCeiling reports ambiguous and never a pick, across BOTH declaration channels. Original text: multiple --max-old-space-size declarations are observable only when every value agrees; divergent values report unknown, never a pick. Config.Cmd does not say which branch is executing, so a pick is a guess with a number attached.

• Any ADR-0025 amendment describes what is actually measured. The wording PR #16634 shipped — "effective ceiling" — encodes the conflation and must not return. — delivered by PR #16883, in two halves. (a) The ADR half is discharged by absence, with a positive control: PR #16634 was closed, never merged, and git log -S"effective ceiling" -- learn/agentos/decisions/ is empty while git log -S"store-ceiling-exhaustion" on the same path returns 82e26297b5 — so the empty result is a finding rather than a broken search. No ADR-0025 amendment exists for this ticket and none is needed: §2.4 governs the routes and thresholds for memory-saturation, never the numerator. (b) The wording was live one file over and is now retired: CEILING_STATE's docblock pointed a consumer at heapSizeLimitBytes as "the effective ceiling" — the exact field the criterion above forbids as a denominator. The paragraph now states what the field is not (the number the process dies at, which is the declaration), names oldGenerationUsedBytes ÷ declaredCeilingBytes as the ratio, and keeps the field because the 48 MiB gap is the evidence the declaration took effect. A guard permits the retired phrasing exactly once, as its own retirement — mutation-convicted: a second live use reddens it. A phrase lint on the decision records was considered and rejected as a syntax proxy with no sunset (ai/deploy/docker-compose.yml:265 uses the phrase correctly).

• Evidence level: residual-live (L3). Unit receipts close a static contract only.

Out of scope

• Lowering the restart-churn threshold from 3 — the code's reasoning for 3 is sound; a heap abort is authoritative at n=1 and was being routed through a frequency signal at all. That is the defect, not the threshold.
• Raising the container limit — the process still self-aborts at V8's heuristic, which does not track the container limit.
• A second "heap" diagnosis kind beside memory-saturation — one signal needs the right numerator, not a sibling free to disagree. This still holds, and the exit-signature work does not violate it — see the note directly below, which resolves the apparent conflict.
• Whether ~560 MiB was a leak or a bounded working set — #16463. Raising a ceiling to hide unbounded growth converts a fast failure into a slow one.
• Service-side ceiling routing and the undeclared-ceiling record — #16636.
• Chroma's ceiling — #16595 / #16596 / #16637.

Related

#16642 / PR #16640 (Slice A, shipped) · PR #16634 (closed unmerged; its review is the authority for this split) · #16636 (act half) · #16459 (the 968-restart loop that produced the orchestrator's declared ceiling) · #16462 (restart-churn signal + the additive ADR-0025 amendment precedent) · #16463 (whether declared ceilings hold) · ADR-0025 / ADR-0026

Origin Session ID: 4141258c-36d3-4788-b0c2-ab3ebe0867be

Retrieval Hint: query_raw_memories("cgroup usage over V8 old-space cap is a cross-scope ratio") · the 11:40:42Z abort · ContainerHealthDiagnosisService.mjs calculateDockerMemoryPercent.