LearnNewsExamplesServices
Frontmatter
id16258
titleWake webhook routes go silent after Memory Core restart
stateClosed
labels
bugaitestingregression
assigneesneo-gpt-emmy
createdAtAug 1, 2026, 2:46 PM
updatedAtAug 1, 2026, 3:26 PM
githubUrlhttps://github.com/neomjs/neo/issues/16258
authorneo-gpt-emmy
commentsCount1
parentIssuenull
subIssues[]
subIssuesCompleted0
subIssuesTotal0
contentTrust
projected
quarantined0
signals[]
blockedBy[]
blocking[]
closedAtAug 1, 2026, 3:26 PM

Wake webhook routes go silent after Memory Core restart

Closed Backlog/active-chunk-11 bugaitestingregression
neo-gpt-emmy
neo-gpt-emmy commented on Aug 1, 2026, 2:46 PM

Context

During the Docker-canonical cutover in #16167, signed a2a-webhook delivery worked before the Memory Core stop/start: the host receiver recorded and visibly delivered a wake. After Memory Core restarted at 2026-08-01T12:04Z, high-priority A2A sends continued to return status: sent and their MESSAGE / SENT_TO / DELIVERED_TO facts appeared in GraphLog, but:

  • Memory Core emitted no coalescing or webhook-delivery log for those events.
  • The host receiver recorded nothing newer than the pre-restart delivery.
  • The durable SQLite WAKE_SUBSCRIPTION row remained active with the same webhook route.

This separates mailbox persistence from last-mile wake delivery. No container mutation was used to diagnose it.

Live latest-open sweep: checked the newest 20 open issues at 2026-08-01T12:43Z; no equivalent current ticket exists. Semantic and historical sweeps found closed #10717 and #11182, which repaired the opposite API/cache split and earlier zero-delivery recurrences. This is a new restart-boundary successor, not a reopening.

The Problem

WakeSubscriptionService.pump() promises to evaluate every active Shape A/B route after a mailbox projection. It calls _warmPushSubscriptions() before selecting active subscriptions, but that helper iterates only GraphService.db.nodes.items.

After a Memory Core restart or restore, durable WAKE_SUBSCRIPTION rows can exist in SQLite without being present in that in-memory graph collection. The pump then sees zero active push routes and advances its live GraphLog cursor past the event. A later caller-owned manage_wake_subscription({action: 'list'}) can hydrate the row, but cannot restore an already-consumed event without an explicit replay.

The result is a healthy mailbox with silently dead webhook wakes precisely after the restart boundary where durable subscriptions are meant to survive.

The Architectural Reality

  • ai/services/memory-core/WakeSubscriptionService.mjs#pump is the low-latency Shape A/B consumer.
  • _warmPushSubscriptions() currently reads the hot graph collection only.
  • _listDurableSubscriptionsForOwner() already demonstrates the SQLite-first parse/hydrate pattern for the public list path.
  • MailboxService._projectMessageWalRecord() invokes the pump only after the durable projection, so the event source is present.
  • Shape C bridge daemons independently tail durable GraphLog/SQLite state and are not the owning substrate for this defect.

The Agent OS structure-map gate was attempted with --files --loc and hit its existing maximum-string failure. The targeted source census still identifies ai/services/memory-core/WakeSubscriptionService.mjs and its existing durable hydration helpers as the owning sibling pattern.

The Fix

Make _warmPushSubscriptions() enumerate active mcp-notifications and a2a-webhook subscriptions from durable SQLite first, hydrate/refresh the hot caches, and invalidate cached push routes whose durable status or target changed. Preserve the current in-memory fallback for isolated tests without raw SQLite storage.

Do this before the pump decides that no active push subscriptions exist or advances its cursor.

Contract Ledger

Target Surface Source of Authority Proposed Behavior Fallback Docs Evidence
Shape A/B push-route census Durable WAKE_SUBSCRIPTION rows in SQLite First post-restart pump sees every active push route In-memory graph scan only when raw SQLite is unavailable Existing JSDoc Cold-cache unit fixture
GraphLog live cursor Pumped durable delta Never advances past a deliverable event merely because the route cache is cold Explicit resync remains recovery for already-consumed history None Cursor assertion + live receiver receipt

Decision Record impact

aligned-with ADR 0002 §6.6.2 — durable wake subscriptions remain the source of truth across restarts. No ADR change.

Acceptance Criteria

  • With an empty in-memory graph/cache and one durable active a2a-webhook row, the first matching pump() enqueues/delivers the event.
  • The same cold-cache contract holds for mcp-notifications.
  • Retired, disabled, and degraded durable routes are not delivered; stale cached routes are refreshed or removed.
  • The pump does not advance its live cursor through a matching event because the push-route cache started cold.
  • Existing cache-hot and no-SQLite test paths remain green.
  • [L4-deferred — operator handoff needed] Post-merge: restart Memory Core, send one high-priority self-message, and record a signed receiver delivery without first calling manage_wake_subscription list.

Out of Scope

  • Shape C bridge-daemon provisioning or GUI/TCC behavior.
  • Digest count/latest-message correctness (#15825 family).
  • Webhook degradation/resume policy (#16246 / #16253).
  • Automatic replay of events already consumed before this fix; use the existing resync surface for recovery.

Avoided Traps

  • Re-subscribing after every restart: hides the durable-source violation and can recreate the duplicate-route class fixed by #10717.
  • Fixing the receiver: the receiver sees no request; the failure is upstream.
  • Reopening a resolved predecessor: this is a new regression ticket linked to the closed history.

Related

Retrieval Hint: WakeSubscriptionService _warmPushSubscriptions durable SQLite cold cache pump advances liveCursor webhook silent after Memory Core restart

tobiu referenced in commit 36a63b7 - "fix(memory-core): warm wake routes from durable state (#16258) (#16260)" on Aug 1, 2026, 3:26 PM
tobiu closed this issue on Aug 1, 2026, 3:26 PM