LearnNewsExamplesServices
Frontmatter
id16750
titleA heap abort records as a generic crash, so the ceiling is never implicated
stateClosed
labels
enhancementaiarchitectureagent-os
assigneesneo-opus-vega
createdAtAug 8, 2026, 11:01 PM
updatedAtAug 9, 2026, 2:06 AM
githubUrlhttps://github.com/neomjs/neo/issues/16750
authorneo-opus-vega
commentsCount0
parentIssuenull
subIssues[]
subIssuesCompleted0
subIssuesTotal0
contentTrust
projected
quarantined0
signals[]
blockedBy[]
blocking[]
closedAtAug 9, 2026, 2:06 AM

A heap abort records as a generic crash, so the ceiling is never implicated

Closed Backlog/active-chunk-14 enhancementaiarchitectureagent-os
neo-opus-vega
neo-opus-vega commented on Aug 8, 2026, 11:01 PM

Context

Split from #16630 Slice B as the honest one-PR leaf. #16630 remains open for the predictive half (a V8-scoped numerator via in-process perf_hooks GC observation) — this ticket delivers only the reactive attribution, which is separable, already implementable, and needs no new channel.

Design work on #16630 this evening produced three measured findings that make this leaf worth filing on its own:

  1. The container-scoped detector cannot fire ahead of a heap death. Both MCP servers run --max-old-space-size=768 under a 1 GiB limit at memorySaturationPercent: 90, so memory-saturation needs 921.6 MiB while old space dies at 768 MiB — it requires ≥153.6 MiB of non-old-space memory to warn at all. Its sensitivity to a heap death is a function of non-heap memory.
  2. There is no IN-PROCESS actuatorv8.setFlagsFromString('--max-old-space-size=N') is accepted and inert (three-arm receipt on #16695, by @neo-opus-grace: reported limit 272 → 272, process aborts identically to the no-flag control). An earlier revision of this ticket generalized that to "no live actuator", which is false and was retracted by its own author: ai/examples/cloud-deployment/deploy-pipeline.sh is a governed recreation channel and has already delivered heap raises in production (#16463 at 2048, PR #16558 persisting 6144). The inert result bounds the in-process layer only.
  3. But the death IS observable after the fact — from the log line, not the exit code. On the canonical neo-local-agent-os-mc-server image (Node v24.16.0, bounded cgroup) a heap death emits FATAL ERROR: … JavaScript heap out of memory and exits 139 with OOMKilled false — in both the declared and undeclared arms (@neo-gpt-emmy). An earlier revision of this ticket recorded 134/0 from a host Node and built a sequencing story on the difference; that was the wrong subject and both claims are retired. The exit code is not even stable across base images, which is the argument against keying attribution to it at all.

Live latest-open sweep: checked the latest 20 open issues at 2026-08-08T20:5xZ; no equivalent found. A2A in-flight claim sweep over the last 30 messages (all read-states): no competing claim on this surface.

The Problem

A service that exhausts its declared V8 ceiling already produces an authoritative container-down fact, already classifies as recoveryClass: 'crash', and is already restarted. The action was never wrong. What is absent is the cause: nothing reads the exit code, so the death records as a generic lifecycle-crash, the ceiling is never implicated, and the identical abort recurs indefinitely.

That is precisely why nothing looks broken from the outside — the heal succeeds every time, and the thing that would tell an operator to raise the ceiling is never emitted.

The Architectural Reality

  • ContainerHealthDiagnosisService.collectLifecycleFacts emits the container-down fact and its details already carry exitCode — the raw signal is present and unread.
  • The crash branch returns reason: 'lifecycle-crash' for every non-running status, regardless of how the process died.
  • DeploymentStateBridgeService independently reads state.OOMKilled, so the field is available at the inspect layer; the diagnosis fact simply does not carry it.
  • #16630's Out of scope forbids "a second heap diagnosis kind beside memory-saturation — one signal needs the right numerator, not a sibling free to disagree." This ticket is an enrichment of an existing authoritative fact, so that line stands unamended.

The Fix

Carry the discriminating evidence on the fact, and narrow the crash reason only when the evidence supports it:

  • classifyHeapExhaustion({logs, nodeCommand, declaredHeapCeilingMb}) — tri-state, returning the attribution plus an unavailableReason.
  • container-down details gain oomKilled (raw evidence) and the attribution fields.
  • The crash branch reports lifecycle-crash-heap-exhaustion[-declared-ceiling] when the evidence names a heap, and lifecycle-crash otherwise. The action class does not change.
  • The bridge summarizes inspect and logs once and passes {logs, nodeCommand, declaredHeapCeilingMb} to diagnose() — no parsing is duplicated in diagnosis.

It is a VERDICT, because the evidence names a heap. An earlier revision emitted a candidate from the exit code — a compromise forced by a capability gap that turned out not to exist. The strict fatal line says heap; nodeCommand says the process had one to exhaust. No abort code carries heap semantics: it names the manner of death only, and every V8 FATAL ERROR, assertion failure and explicit abort() shares it.

Tri-state is load-bearing: null means the deciding fields were not observed and must read as absence of signal, never as a negative. A missing ExitCode is not evidence of a healthy exit.

Contract Ledger Matrix

Target Surface Source of Authority Proposed Behavior Fallback Docs Evidence
ContainerHealthDiagnosisService container-down fact details this ticket; existing fact already carries exitCode gains oomKilled as raw evidence + {heapExhaustion, unavailableReason, declaredHeapCeilingMb} unobservable → null with a reason, never a negative JSDoc on the helper unit spec, every unavailable reason
classifyHeapExhaustion({logs, nodeCommand, declaredHeapCeilingMb}) (new export) this ticket attribution from the strict fatal line and nodeCommand null + unavailableReason when the question cannot be asked JSDoc unit spec incl. the non-Node red control
HEAP_FATAL_LINE (new export) V8's own fatal wording, both shapes strict single-line pattern JSDoc unit spec matches both V8 shapes
diagnose() inputs DeploymentStateBridgeService :392 / :1122-1123 accepts {logs, nodeCommand, declaredHeapCeilingMb} from the bridge's existing observations null each, degrading to unavailable JSDoc bridge + diagnosis unit specs
crash diagnosis classificationReason ADR-0026 §2.4 actuator matrix narrows to lifecycle-crash-heap-exhaustion[-declared-ceiling]; action class unchanged generic lifecycle-crash unit spec asserts the action is unchanged
DeploymentRuntimeAccessService.readObserve({operation:'logs'}) this ticket; existing allowlisted read-observe operation accepts since / until; narrows an existing read, adds no capability, operation, or privilege either bound missing/invalid ⇒ unbounded query, bounded: false JSDoc producer spec: interval in the URL + echoed receipt
logs read receipt (data) same gains appliedSince / appliedUntil at full RFC3339 precision + containerId null each; bounded:false JSDoc spec asserts sub-second endpoints survive unrounded
summarizeLogs()logs.incarnationBounded DeploymentStateBridgeService true only from the producer receipt and a logs/inspect target match false — never inferred, never caller-supplied inline spec: a different-container read never publishes bounded

Decision Record impact

aligned-with ADR 0026 — the §2.4 apply(serviceKey, action) interface and the detect ÷ actuator separation are untouched; no action class is added, removed, or re-routed. aligned-with ADR 0025 — this enriches a diagnosis fact rather than changing what the diagnosis decides.

Acceptance Criteria

Rewritten 2026-08-08 ~21:5xZ. The original criteria specified an exit-code discriminator (ExitCode === 134). That was falsified during review: the canonical neo-local-agent-os-mc-server image exits 139 where a host Node exits 134, and it exits 139 whether or not a ceiling was declared — so the code neither ports nor discriminates. The criteria below specify the evidence that actually names a heap. Falsification and the B-prime scoping correction: @neo-gpt-emmy on PR #16751.

  • Attribution requires both the strict fatal line (FATAL ERROR: … JavaScript heap out of memory) and nodeCommand === true. Either alone must not attribute.
  • The declared ceiling is not part of the discriminator. An undeclared Node service that dies of a heap still attributes — that population is where this class was first observed, and scoping to the ceiling would blind the detector to it. A numeric declaredHeapCeilingMb licenses only the stronger wording.
  • A non-Node service never attributes, whatever its log tail contains — proven by a red control, since a tail can carry another container's output or a service can simply log the phrase.
  • Unavailable is null with a stated reason, never false: logs disabled, an unreadable command, and a truncated non-matching tail each yield null plus an unavailableReason. A disabled channel is not evidence of health.
  • A truncated tail that DOES match stays conclusive — truncation cannot manufacture the line.
  • The container-down fact carries exitCode and oomKilled as raw evidence alongside the attribution — recorded, but not attribution inputs.
  • The crash diagnosis reports lifecycle-crash-heap-exhaustion-declared-ceiling, lifecycle-crash-heap-exhaustion, or lifecycle-crash, matching the evidence available.
  • The action class is unchanged in every branch — attribution must not alter the heal.
  • The scoping guard is shown capable of failing: removing the nodeCommand requirement reds the non-Node red control.
  • No parsing is duplicated in diagnosis. nodeCommand and declaredHeapCeilingMb are consumed from the bridge's existing Config.Cmd observations, and the log tail from its existing allowlisted read.
  • The log slice is bound to the stopped incarnation by an EXACT interval. since = State.StartedAt, until = State.FinishedAt, both transmitted unrounded — flooring the upper edge discards the final sub-second in which V8 writes its fatal line, and flooring the lower edge reaches into the previous run.
  • The bound proves its TARGET, not just a time range. readObserve resolves a target per call, so incarnationBounded requires the logs read and the inspect to have resolved the same container. A matching range on the wrong container is not this incarnation.
  • Either bound missing, invalid, or half-open ⇒ unbounded, and the classifier keeps refusing with log-incarnation-unbounded. Docker's unset time is the zero instant, which parses valid — it must be refused rather than passed through.

Out of Scope

  • The V8-scoped numerator (#16630 Slice B AC-1) — the predictive perf_hooks GC-observer channel. This ticket delivers attribution, not saturation.
  • Any change to memory-saturation or its threshold — the numerator moves scope there, not here.
  • A new log-read capabilityretired as false. DeploymentStateBridgeService:392 already performs the allowlisted read and :449 publishes a bounded tail, so no new capability or privilege is involved. Asserting this boundary without checking it is what justified the weaker exit-code design.
  • Raising a ceiling in response#16695 establishes only that no in-process channel exists; a governed recreation channel does, and the actuator question is @neo-opus-grace's lane.

Avoided Traps

  • Attributing on the exit code at all. An earlier revision of this ticket said 134 was "necessary and not sufficient" — the canonical image falsified even the necessary half (it exits 139, in both the declared and undeclared arms). No abort code carries heap semantics: it names the manner of death, and its value drifts with the base image. A detector keyed to one would both misattribute other abort classes and miss the real thing.
  • Matching the fatal line anywhere in the tail. The bounded log read spans container restarts, so an old heap-fatal line above a healthy boot above an unrelated crash attributes the current death as heap. The evidence window must be bound to the stopped incarnation — the same wider-than-the-thing-it-names defect #16500 was opened for.
  • A boolean instead of a tri-state. false for an unobserved field would assert "not a heap abort" from missing data, which is the fabricated-verdict shape the presence-contract work on D#16720 rejects on a different substrate.
  • A new fact type. A sibling diagnosis free to disagree with memory-saturation is exactly what #16630 excludes; enrichment keeps one signal.
  • Changing the action. Restart is already correct for a stopped container; a "fix" that alters the heal would be trading a real repair for a label.

Related

#16630 (parent scope — Slice B's predictive half stays open there) · #16695 (no in-process V8-ceiling actuator; a governed recreation channel exists — @neo-opus-grace) · #16636 (service-side ceiling routing) · #16642 / PR #16640 (Slice A, which produced the signature this reads) · ADR-0025 / ADR-0026

Origin Session ID: 46485b3a-9468-4df8-9390-3c961bd0e72f

Retrieval Hint: query_raw_memories("heap abort exit 134 SIGABRT candidate not verdict container-down attribution undeclared ceiling exits 0 no signature")

Authored by Vega (@neo-opus-vega, Claude Opus 5, Claude Code).

tobiu closed this issue on Aug 9, 2026, 2:06 AM
tobiu referenced in commit d589d8c - "feat(orchestrator): the crash diagnosis names a heap abort instead of a generic crash (#16750) (#16751) on Aug 9, 2026, 2:06 AM