LearnNewsExamplesServices
Frontmatter
id16776
titleHeap-observation channel: collector, cadence reporter, bridge reader
stateClosed
labels
enhancementaiarchitectureagent-os
assigneesneo-opus-vega
createdAtAug 9, 2026, 12:14 PM
updatedAtAug 9, 2026, 12:38 PM
githubUrlhttps://github.com/neomjs/neo/issues/16776
authorneo-opus-vega
commentsCount0
parentIssue16763
subIssues[]
subIssuesCompleted0
subIssuesTotal0
contentTrust
projected
quarantined0
signals[]
blockedBy[]
blocking[]
closedAtAug 9, 2026, 12:38 PM

Heap-observation channel: collector, cadence reporter, bridge reader

Closed Backlog/active-chunk-14 enhancementaiarchitectureagent-os
neo-opus-vega
neo-opus-vega commented on Aug 9, 2026, 12:14 PM

Related: #16763

Context

This is the implementation leaf split out of #16763, and the split is a repair of a defect in that ticket's own shape rather than a convenience.

ticket-create-workflow.md §4 requires a standalone ticket to be one-PR-resolvable — one Resolves, so a second PR closes it early or fails the lint. #16763 is not: its AC-9 states that "a live reading from the running deployment is required to close the ticket", which no implementation PR can supply, because the running revision never contains the code being merged. Two completion states were therefore packed into one ticket: the mechanism ships and the mechanism is observed working. This ticket owns the first. #16763 stays open and owns the second.

Reviewer @neo-gpt-emmy identified the close-target defect on PR #16771 (review PRR_kwDODSospM8AAAABI4Dy7A) and named this split as one of two acceptable dispositions. The other — a non-closing Related: #N on the implementation PR — is mechanically unavailable: agent-pr-body-lint requires Resolves #N on any non-draft agent PR, and Refs/Related stand in only for drafts (buildScripts/util/agent-preflight.mjs:29-31, 228-231). Marking a code-complete PR draft to satisfy a linter would block its review instead.

Live latest-open sweep: checked the latest 20 open issues at 2026-08-09T10:20Z; no equivalent found. A2A in-flight claim sweep over the last 30 messages: no claim on this scope.

The Problem

A docker stats reading is container RSS — one number fusing V8 old space, new space, native allocations, Buffers and the binary. --max-old-space-size bounds the old generation alone. Comparing the first against the second is a cross-scope ratio that can read as authoritative on memory V8 never touched, and can miss a real old-space exhaustion. No Docker read operation can fix it, because no read operation can see inside a process. Only the process can answer.

The full framing, the measurements that shaped the design, and the saturation question this feeds live on #16763 and are not restated here.

The Architectural Reality

  • ai/services/shared/processHeapObservation.mjs — the synchronous collector. One timestamp, every source read inside one block.
  • ai/mcp/server/shared/services/HeapObservationReporterService.mjs — the cadence writer, write-then-rename into the shared plane root.
  • ai/mcp/server/BaseServer.mjs — the lifecycle owner. initAsync() is the single point every server converges on: subclasses with a non-canonical bootstrap override boot() without chaining super.boot() (ai/mcp/server/memory-core/Server.mjs:272 does exactly that), so a start wired into the default boot() skips the servers most worth observing.
  • ai/daemons/orchestrator/services/DeploymentStateBridgeService.mjs — the reader, bounding freshness and skew before publishing.
  • ai/configBase.mjsheapObservation at Tier-1, because writer and reader are different processes reading the same leaves. heapObservation.dir is a PLANE_MEMBER_PATHS member.

The per-server config bases (ai/mcp/server/{knowledge-base,memory-core}/configBase.mjs) extend ConfigProvider directly, not the Tier-1 ConfigBase — so this.aiConfig.heapObservation does not resolve on a server instance, and the Tier-1 AiConfig use-site read is the correct source per ADR-0019.

The Fix

Delivered by PR #16771. The observation channel end to end: collector, reporter, boot ownership with teardown, bridge reader, config leaves, and the plane-member placement.

Contract Ledger Matrix

Target Surface Source of Authority Proposed Behavior Fallback Docs Evidence
collectProcessHeapObservation() #16763 heap + non-heap captured at one instant, one timestamp throws never — including the clock and the argument vector; unavailability is an unavailableReason string JSDoc single-instant mutation falsifier; clock-throw and hostile-argv falsifiers
readDeclaredCeiling() measured Node behavior reads the ceiling from process.execArgv and NODE_OPTIONS divergence is ambiguous, never a resolved pick JSDoc spec: ceiling declared only via NODE_OPTIONS; cross-channel divergence
HeapObservationReporterService #16763 writes the record to the shared plane root on a cadence total: target resolution, write/rename, injected logger and staging cleanup each return false rather than escaping JSDoc write-, logger-, cleanup-, target- and config-failure arms
BaseServer.getHeapObservationServiceKey() this ticket opt-in declaration of a Compose service label; null publishes nothing a start failure never blocks boot JSDoc boot witness + a red control for a server declaring no key
DeploymentStateBridgeService per-service record existing providerResidency precedent gains nullable heapObservation null with a reason: absent, stale, skewed, non-Node, identity-mismatch, clock-skew JSDoc bridge spec, every null reason
memory-saturation diagnosis ADR-0025 / ADR-0026 unchanged spec asserts the fact is byte-identical with and without the field

Decision Record impact

aligned-with ADR 0019, aligned-with ADR 0025, aligned-with ADR 0026. Config leaves are declared at Tier-1 and read at use-site; the shared AiConfig singleton is never mutated. No diagnosis changes what it decides and the §2.4 actuator matrix is untouched.

Acceptance Criteria

  • The pair is captured at one instant — a mutation converting the collector to async with one await between source reads reds the guard. A call-count assertion would pass that mutation.
  • heap_size_limit is recorded as observed, never derived. The observed limit and the declared ceiling are separate fields and no constant relates them in code.
  • Both declaration channels are read. Node does not merge NODE_OPTIONS into process.execArgv; reading execArgv alone reports undeclared for a ceiling genuinely in force. Divergence between channels is ambiguous.
  • Unavailable is null with a stated reason, never false and never 0.
  • The reporter is total. Target resolution, the write, the injected logger and staging cleanup each yield false. A service must not die because it could not describe its own heap.
  • The channel is reachable from production boot, proven without calling the reporter. A witness boots a server through its ordinary lifecycle and asserts the file appeared at the configured path, with a red control for a server declaring no key. A witness that constructs the singleton directly proves only that the class exists.
  • A dead reporter never reads as healthy — the bridge surfaces null + stale, never the last-known value.
  • Skew is bounded against the paired container reading.
  • A non-Node service never produces an observation, proven by a red control.
  • No fact is emitted and no threshold moves — the memory-saturation fact is asserted unchanged with the field present and absent.
  • Evidence level: L2. Unit receipts plus the collector executed inside the shipped image under a real cgroup close this ticket. The live-deployment reading is #16763's AC-9 and is deliberately not an AC here.

Out of Scope

  • The live L3 reading#16763's AC-9. That is the whole reason this ticket exists as a separate leaf.
  • Computing saturation from the numerator#16630 Slice B. This makes the number observable; deciding what it means stays there.
  • A PLANE_MEMBER_PATHS ↔ profile x-plane-env cross-check. Nothing mechanically verifies this. Corrected: test/playwright/unit/ai/deploy/ParityPlaneVolumeScoping.spec.mjs:317 already does, shipped in #16761 before this branch existed — removing the binding reds it with heapObservation.dir (NEO_HEAP_OBSERVATION_DIR) is absent from x-plane-env. #16777 was filed against this false premise and is closed as a duplicate of shipped work.

Avoided Traps

  • Resolving the CLI-vs-NODE_OPTIONS precedence in this module. Measured on node v25.9.0, one rule fits all five cases (concatenate NODE_OPTIONS then the command line, take the last), but it is V8's rule, not ours, and heapSizeLimitBytes is already observed independently — so a consumer needing the effective ceiling has it from the instrument. Restating the rule here would mean keeping it in sync with a runtime we do not control.
  • An HTTP endpoint instead of the file transport. It would add a network dependency and a second runtimeMechanism to a proof model where every existing record declares docker-socket. The falsifier for this choice: a staleness bound cannot distinguish a slow reporter from a dead one, while a request/response can.
  • Substituting Date.now() when the injected clock throws. That would stamp the record with a clock the caller did not choose. observedAt: null degrades to the reader's malformed arm — absence, never a wrong age.

Related

  • #16763 — parent question; stays open for the live L3 receipt.
  • #16630 — the saturation semantics that will consume this.
  • PR #16771 — the delivering PR.

Origin Session ID: 4131135d-1b20-487f-9d23-d7213914246b

Retrieval Hint: query_raw_memories("self-reported heap observation channel boot reachability"); commit range 464233a19a..HEAD on agent/16763-heap-observation-channel.

tobiu closed this issue on Aug 9, 2026, 12:38 PM