LearnNewsExamplesServices
Frontmatter
id17691
titleThe nightly e2e runner can detect a red and still tell nobody — three independent silences
stateClosed
labels
bugaitestingagent-os
assigneesneo-opus-grace
createdAtAug 24, 2026, 11:18 AM
updatedAtAug 24, 2026, 5:28 PM
githubUrlhttps://github.com/neomjs/neo/issues/17691
authorneo-opus-grace
commentsCount0
parentIssuenull
subIssues[]
subIssuesCompleted0
subIssuesTotal0
contentTrust
projected
quarantined0
signals[]
blockedBy[]
blocking[]
closedAtAug 24, 2026, 5:28 PM

The nightly e2e runner can detect a red and still tell nobody — three independent silences

Closed Backlog/active-chunk-19 bugaitestingagent-os
neo-opus-grace
neo-opus-grace commented on Aug 24, 2026, 11:18 AM

⚠️ Amended 2026-08-24 after review falsified defect B's remedy. @neo-gpt-emmy showed that "publish the receipt on the Memory Core healthcheck" has no carrier: canonical mc-server mounts only sqlite, handoff, deployment-state, vector-generation and heap-observation — no nightly-e2e — so a relative path resolves inside the container layer, never to the host LaunchAgent's receipt. It would have reported unavailable permanently, from transport, which is the exact value the field exists to disambiguate. The plane-literal ledger acquittal established ownership; it never established reachability. A matching pathname is not a channel.

Defect B's diagnosis stands and is unchanged — the receipt is written every run and read by nobody. Only its remedy is withdrawn from this leaf, because designing a host→container carrier is a separate architectural question and not a patch. The compose file's own heap-observation mount comment is the precedent for what a real carrier looks like, and states this failure verbatim: "Without a shared mount the reporter writes into the container's own layer and the orchestrator reads its own empty one, so every observation surfaces as unavailable."

This leaf now owns defects A and C only. B is re-homed to #17708, which also corrects this amendment's "no carrier" framing: the runner holds an in-process Memory Core service connection it already uses to deliver the digest, so the design space is narrower than stated here.

Context

Surfaced while dispositioning #17596's activation preconditions. The operator asked the question that opened this: "e2e runner => how would our ai team know in case it fails?"

ai/scripts/lifecycle/nightlyE2eRunner.mjs is the host-side observer for the whitebox-e2e layer — deliberately outside CI, because e2e needs GPU hardware (playwright.config.e2e.mjs:49 pins branded Chrome, :92 gates on it resolving to real hardware). Its JSDoc calls it "the unattended quality heartbeat" whose "one job is to not go silent."

It has never run on the canonical host (no com.neomjs.nightly-e2e in launchctl, no plist, no state dir), so none of the below has ever been observed. These are defects of the reporting path, independent of activation and of the schedule.

The Problem

Three independent ways the runner can find a red and fail to inform anyone. Each was verified at dev.

A. The RED digest does not wake anyone

await MailboxService.addMessage({
    to      : 'AGENT:*',
    subject : `[nightly-e2e][RED] …`,
    priority: 'normal'
});

No wakeSuppressed is passed. MailboxService.mjs:2446 resolves wakeSuppressed = wakeSuppressed ?? (operatorSteering || to === 'AGENT:*') — so an AGENT:* broadcast defaults to suppressed. The safety guard does not object either: isAllowedWakeSuppression returns true unconditionally for AGENT:*.

So the digest lands silently in mailboxes that routinely carry 2,000+ unread. A red suite is action-required by definition, and it arrives as drain-class.

This is the runner's half of a defect class #15919 already owns ("Wake: AGENT:* quiet-by-default with a derived structural attention set") — that ticket is the general repair, where a broadcast can be durable-to-all while waking only the actors. This leaf must not re-solve it; it should use the lever available today.

B. The liveness receipt exists and reaches no surface

STATE_PATH = '.neo-ai-data/nightly-e2e/last-run.json' is written on every run, red or green, carrying {at, red, configs}. So the data needed to distinguish outcomes exists.

Nothing reads it. A search across ai/, src/, apps/ finds only the plist and README naming that directory — no consumer, no healthcheck field, no digest.

The runner's contract is "Green = silence." Combined with an unpublished receipt, absence of a digest means one of three things — all green, the runner never fired, or it crashed before reporting — and a reader cannot tell which. That is exactly the state the layer has been in: 17 red specs and total silence, indistinguishable from health.

This is not a missing measurement; it is a measurement that never reaches a decision.

C. A failed digest send loses the red entirely

try {
    …
    await LifecycleService.ready();
    await GraphService.ready();
    await RequestContextService.run(…, async () => { await MailboxService.addMessage({…}) });
} finally {
    await fs.remove(LOCK_PATH).catch(() => {});
}

There is no catch. If ready() or addMessage throws — Memory Core unreachable, Chroma down, identity unbound (healthcheck reported identity.bound: false on this host on 2026-08-24) — the exception propagates, the detected red is never reported, and the finally releases the lock cleanly so the next run starts with no record that anything was found.

The reporter's failure mode is identical to its success mode.

The Architectural Reality

  • nightlyE2eRunner.mjs already owns the state dir, the lock with a 6h staleness steal, the per-run log, and last-run.json. No new file or directory is needed.
  • It already treats no-parseable-report + non-zero exit as infra-red the digest must surface rather than swallow — the same discipline this ticket extends to the digest send itself.
  • The Memory Core healthcheck is the natural publication surface for B: it already carries maintenance.* and backup observability blocks, and #17495 established the vocabulary for distinguishing observed from unavailable rather than reporting a zero.
  • Activation remains operator-owned (#17596) and is not a precondition for any of this: all three repairs are correct whether or not the plist is ever installed.

Contract Ledger Matrix

Target Surface Source of Authority Proposed Behavior Fallback / Edge Case Docs Evidence
RED digest wake tier a red suite is action-required the digest opts out of suppression explicitly (wakeSuppressed: false) green stays silent and un-woken; #15919 supersedes this with a derived attention set when it lands runner JSDoc unit arm asserting the red path passes explicit non-suppression
Run liveness last-run.json, already written every run published where a reader looks, so absence-of-digest is decidable never-run reports as unavailable, never as green; a stale receipt reports its age rather than a verdict runner JSDoc + the consuming surface arms for never-ran, green, and red
Digest-send failure the reporter's own infra-red discipline a failed send is itself reported — durable local record plus non-zero exit the detected red survives the reporting failure and is visible to the next run runner JSDoc arm injecting an addMessage throw and asserting the red is not lost

Decision Record impact

none. No ADR surface: this repairs a script's reporting path and consumes an existing healthcheck vocabulary.

Acceptance Criteria

  • A RED digest is emitted with explicit non-suppression, so it reaches its recipients as attention rather than drain — asserted on the addMessage payload, not on a log line.
  • A GREEN run remains silent and wakes nobody; the change must not convert routine success into traffic.
  • Delivery disposition is recorded, never derived: the receipt carries an explicit state written before the attempt, and a missing field can never render as delivered.
  • A throwing digest send does not lose the detected red: the failure is recorded durably and the process exits non-zero.
  • A crash between receipt-write and send leaves the unresolved state standing, never a delivered one.
  • The lock is still always released, including on the new failure path.
  • Red-proof: reverting each repair independently turns its own arm red and no other.

Out of Scope

  • Activating the LaunchAgent or choosing its schedule — operator-owned, tracked on #17596.
  • Publishing run liveness to any reader — withdrawn to successor leaf #17708. This leaf asserted it needs a designed host→container carrier; #17708 establishes that the runner already owns a Memory Core service connection, so reachability must be measured before any transport is chosen.
  • The general AGENT:* attention-set redesign — #15919 owns it; this leaf uses the existing per-message lever.
  • Fixing any of the 17 red e2e specs — #17596 owns the triage.
  • Adding e2e to CI, which is unbuildable (GPU-gated by construction).
tobiu referenced in commit 0db7ba4 - "fix(lifecycle): the nightly e2e runner can no longer find a red and tell nobody (#17691) (#17693) on Aug 24, 2026, 5:28 PM
tobiu closed this issue on Aug 24, 2026, 5:28 PM