LearnNewsExamplesServices
Frontmatter
id17714
titleThe nightly e2e digest is written in-process to a store no reader serves
stateClosed
labels
bugaitestingagent-os
assigneesneo-opus-grace
createdAtAug 24, 2026, 6:11 PM
updatedAtAug 24, 2026, 7:16 PM
githubUrlhttps://github.com/neomjs/neo/issues/17714
authorneo-opus-grace
commentsCount0
parentIssuenull
subIssues[]
subIssuesCompleted0
subIssuesTotal0
contentTrust
projected
quarantined0
signals[]
blockedBy[]
blocking[]
closedAtAug 24, 2026, 7:16 PM

The nightly e2e digest is written in-process to a store no reader serves

Closed Backlog/active-chunk-19 bugaitestingagent-os
neo-opus-grace
neo-opus-grace commented on Aug 24, 2026, 6:11 PM

Context

Split from #17708, which measured the plane and found it unshared in both directions. That leaf owns the liveness publication question — how a reader learns the runner ran at all. This one owns the narrower, independently deliverable half its AC-1 measurement exposed: the RED digest does not arrive either, and for the same reason.

Split because #17708 grew past one-PR-resolvable once its own measurement widened it. The digest transport is a self-contained change with its own arms; the liveness surface is a separate design.

The Problem

ai/scripts/lifecycle/nightlyE2eRunner.mjs delivers its digest through in-process Memory Core service imports:

import GraphService          from '../../services/memory-core/GraphService.mjs';
import LifecycleService      from '../../services/memory-core/lifecycle/SystemLifecycleService.mjs';
import MailboxService        from '../../services/memory-core/MailboxService.mjs';
import RequestContextService from '../../mcp/server/shared/services/RequestContextService.mjs';

Those resolve against the host plane root. Measured on the live deployment (#17708 AC-1): the host and the canonical containerized mc-server hold two different SQLite stores. Three A2A messages sent through MCP during that session — which peers received and replied to — were absent from the host store by exact primary-key lookup, against a positive control of 12273 MESSAGE:% rows proving the selector matched. The plist supplies no environment that would redirect it.

So the digest write succeeds locally and arrives nowhere. #17691's repair made the RED digest wake its recipients; this is the layer under it — the wake tier of a message that never reaches the store anyone reads.

The runner has never been installed on the canonical host, so this is a prediction from where the write lands rather than an observed lost digest. That distinction is why this is filed as its own leaf: it is falsifiable by a first run, and it should be fixed before that run rather than after.

The Architectural Reality

The in-process idiom is correct for its siblings and wrong only here, which is why it survived review:

  • ai/daemons/orchestrator/services/SwarmHeartbeatService.mjs:36 consumes swarmWakeCooldown, and :154 calls MailboxService.sweepExpiredTasks() directly. Both run inside the orchestrator container, where an in-process import resolves against the plane that container serves. That same file records sweepExpiredTasks was deliberately moved inward from a host subprocess.
  • nightlyE2eRunner.mjs is the only member with a host LaunchAgent plist, and it cannot follow them inward: playwright.config.e2e.mjs:49 pins branded Chrome and :92 gates on real GPU hardware. Being outside the container is the reason the script exists.

It inherited an idiom from siblings that no longer share its execution context.

ai/mcp/client/config.mjs already declares the memory-core client entry (streamable-http, local ingress, bearer credential), and ai/mcp/client/Client.mjs is used host-side by kbPushClient.mjs, communityBatchPushClient.mjs and ai/Agent.mjs. The transport exists; only this consumer is missing.

The Fix

Deliver the digest as an authenticated MCP client of the containerized Memory Core.

  1. Replace the four in-process service imports with Client against the declared memory-core entry.
  2. Collapse the addMessage / graphReady / lifecycleReady seams into one connect seam. Readiness hooks exist to bootstrap in-process services; a client does not bootstrap a server. A connection failure and a rejected call stay distinguishable because they arise at different calls.
  3. Validate the credential before Neo.create. create() runs initAsync in a detached promise with no rejection handler (src/core/Base.mjs:314), so a throw there never reaches ready() — the promise never resolves and an unattended run would hang until the 6h stale-lock steal.
  4. Declare EnvironmentVariables on the plist template. A launchd session inherits nothing from a shell, so without it the run reaches the client with no credential and no node on PATH.
  5. Record the sender-identity change: the literal @system is replaced by whatever the credential resolves to, so the README must direct an automation identity rather than a maintainer seat.

Contract Ledger

Target Surface Source of Authority Proposed Behavior Fallback / Edge Case Docs Evidence
Digest delivery transport #17708 AC-1 measurement; ai/mcp/client/config.mjs memory-core entry An add_message tool call against the containerized Memory Core over the declared entry Unreachable/unauthenticated throws; the receipt records failed and the process exits non-zero Runner JSDoc + nightly-e2e README Arms over RED send, GREEN silence, connect failure, call failure
Runner collaborator seam Existing injectable seams One connect seam returning {callTool, close} A stub client drives every arm; no network in tests Runner JSDoc Existing delivery arms re-pointed at the seam
Credential availability requiredEnv on the memory-core entry Validated before client construction so failure is loud Missing credential throws a named error citing the README Runner JSDoc + README install step A no-credential arm asserting the throw and a failed receipt
LaunchAgent environment Sibling ai/deploy/com.neomjs.agent-os-*.plist templates EnvironmentVariables carrying PATH + the credential, via the same __PLACEHOLDER__ substitution Rendered copy holds a secret; README sets chmod 600 README activate step Install step renders and loads

Decision Record impact

none. Consumes an existing declared client entry and an existing plist convention; introduces no new transport, surface, or config leaf.

Acceptance Criteria

  • The runner imports no Memory Core service module; the digest travels as an add_message tool call over the declared memory-core client entry.
  • The collaborator seam is a single connect returning {callTool, close}; a connect failure and a call failure are separately asserted and separately recorded.
  • The asserted contract includes the tool name, not only the payload, so a rename on either side fails a test.
  • A missing credential throws before client construction, and the run records failed rather than hanging or reporting success.
  • The plist template declares EnvironmentVariables with PATH and the credential placeholder; the README install step renders both and restricts the rendered copy.
  • The sender-identity change is stated in the README, directing an automation identity rather than a maintainer seat.
  • The transport closes on every path, including both failure paths.
  • Red-proof: reverting each repair independently turns its own arm red and no other.

Out of Scope

  • Publishing run liveness to any reader — #17708 owns it, and it is a separate design.
  • The host-store retirement gate — #17708 owns it, cross-pinned with #16202.
  • Activating the LaunchAgent or choosing its schedule — operator-owned, tracked on #17596.
  • Fixing any red e2e spec — #17596 owns that triage.
  • Changing Memory Core's ingress, auth mode, or tool surface.

Avoided Traps

  • Add a nightly-e2e mount to mc-server. Rejected: it shares a file and does nothing for the digest, which travels through MailboxService, not last-run.json. It would fix the smaller half and add deployment topology for a process that already has a working client path.
  • Move the runner into the orchestrator like its siblings. Rejected: GPU- and headed-browser-bound by construction.
  • Keep await client.initAsync() after Neo.create. Rejected — valid API, wrong idiom. create() already fires initAsync detached, so the explicit call double-initializes and its throw escapes as an unhandled rejection instead of failing the run.
  • Rely on the client's own requiredEnv check for loudness. Insufficient alone: it throws inside the detached initAsync, where ready() never settles. The pre-construction check is what keeps the failure loud.
  • Treat "the digest reaches a mailbox in an interactive shell" as proof. Rejected: an interactive shell exports the credential and launchd does not, so the only passing context is the one the runner never runs in.

Related

Parent measurement + successor scope: #17708
Predecessor (wake tier + send-failure repairs): #17691, PR #17693 (merged)
Activation owner: #17596
Source module: ai/scripts/lifecycle/nightlyE2eRunner.mjs

Live latest-open sweep: checked latest 20 open issues at 2026-08-24T16:09Z; A2A all-state claim sweep over the most recent 12 messages at the same time; no equivalent ticket and no competing claim.

Origin Session ID: 728a756d-71df-48e6-8dad-0bac498ca23e

Retrieval Hint: query_raw_memories("nightly e2e runner in-process MailboxService host plane MCP client transport")
Retrieval Hint: nightlyE2eRunner host store vs container store; siblings run in-container via SwarmHeartbeatService

tobiu referenced in commit 6981af4 - "feat(lifecycle): the nightly runner delivers as an MCP client (#17714) (#17715) on Aug 24, 2026, 7:16 PM
tobiu closed this issue on Aug 24, 2026, 7:16 PM