Wake delivery cannot degrade a dead route, so it retries forever

Context

Observed live in the containerized Memory Core on 2026-08-01, triggered by an ordinary add_message broadcast. From /app/.neo-ai-data/logs/mc-server-2026-08-01.log inside neo-local-agent-os-mc-server-1:

00:07:25 [ERROR] WebhookDeliveryService: Network error delivering to WAKE_SUB:36342c42-…: fetch failed. Attempt 1/4
00:07:26 [ERROR] … Attempt 2/4
00:07:28 [ERROR] … Attempt 3/4
00:07:35 [ERROR] … Attempt 4/4
00:07:35 [WARN]  WebhookDeliveryService: 3 consecutive failures for WAKE_SUB:36342c42-…. Marking degraded.
00:07:35 [WARN]  WebhookDeliveryService: Cannot degrade WAKE_SUB:36342c42-…, node not found in Graph.

The delivery failure itself is expected and not the subject of this ticket — no wake receiver is currently listening, which is #16233's scope. The defect is the last line: the service decided to degrade the route and could not, so nothing changed and the next message repeats the whole sequence.

Observation separated from inference: the log lines above are observed. The mechanism in The Problem is derived from source plus a within-run control, and its necessity is not yet directly measured — the falsifier is named in the ACs.

The Problem

The node is not missing. The read path cannot see it.

The within-run control is decisive: in that same flush, the same process successfully read the same subscription — it had to, in order to resolve the url and signingKey it delivered against. One read succeeded and one returned nothing, milliseconds apart, on the same node. So "node not found" is not about the node's existence.

The two reads use different APIs:

• WakeSubscriptionService reads through GraphService.db.nodes.get(id) directly, with an explicit comment at ai/services/memory-core/WakeSubscriptionService.mjs:854 — "Access GraphService.db.nodes.get directly because GraphService.getNode filters out custom properties" — and again at :1373.
• WebhookDeliveryService._markDegraded calls GraphService.getNode({id: subscriptionId}) at ai/services/memory-core/WebhookDeliveryService.mjs:158.

GraphService.getNode applies a row-level-security re-check at its return boundary (GraphService.mjs:845-867). isRlsVisible (:27-45) returns true only when the owner key is null, the normalized owner matches the requester, the entity is shared, or visibility is team. resolveRlsUserId (:62-65) resolves the requester from the request-bound RequestContextService.

The coalescing flush is a background timer, not an MCP request. With no bound request context there is no requester id, the subscription is owner-stamped and not shared, and every branch of isRlsVisible evaluates false — so getNode returns null and the degrade is skipped. WakeSubscriptionService already knows this and routes around it; WebhookDeliveryService does not.

Falsified competing hypothesis. Stored userId is inconsistent across subscription vintages — some records carry @neo-opus-ada, others neo-opus-ada. That looked like the cause and is not: isRlsVisible normalizes the owner key before comparing (GraphService.mjs:41), explicitly to match an agent's own nodes regardless of stored form. Recording it so the next reader does not re-derive and stop there.

Consequence. A permanently dead route is never marked degraded, so consecutiveFailures keeps climbing past 3 and re-fires _markDegraded on every subsequent message, each time preceded by 4 delivery attempts with backoff (:145-153). Every A2A message to that seat costs four failed network round trips forever. The only signal is a WARN in a file nobody tails — docker logs does not carry it.

The Architectural Reality

This is a service-boundary seam, not a logic bug in either component. getNode's RLS is behaving correctly and must not be weakened; the caller is simply on the wrong side of a request boundary. The fix belongs to the caller.

The Fix

1. _markDegraded reads the subscription through the same context-free path its sibling uses, rather than the request-scoped getNode. GraphService.getNodeRecord-style properties-returning reads exist (GraphService.mjs:869-879) but carry the identical RLS re-check, so they are not a fix — the background-writer path is what needs naming.
2. Give the background writer an explicit, named accessor rather than each service reaching into GraphService.db.nodes ad hoc. Two call sites already do it by hand with an explanatory comment; a third arriving by copy-paste is how this recurs.
3. _recordConsecutiveFailure must not re-enter the 4-attempt delivery cycle for a route already known-dead. Today the threshold triggers a degrade that silently fails, and nothing caps the repetition.
4. Decide and document what harnessTarget: 'degraded' is. It is written at :162 but appears in no enum: the manage_wake_subscription tool schema accepts only mcp-notifications | a2a-webhook | bridge-daemon | disabled | none. It currently "works" only by falling through CoalescingEngineService's unknown-target branch, which drops the digest with a WARN. Working by accident through an unknown-value fallback is not a state model — either add it to the enum as a first-class state, or record degradation outside harnessTarget and leave the routing field to routing.

Contract Ledger

Decision Record impact

none. This restores intended behaviour at a service seam; it does not alter the RLS model, which is working as designed.

Acceptance Criteria

• A subscription whose endpoint refuses connections reaches a degraded state, verified by reading the subscription back after the failure threshold — not by reading the log line.
• The necessity probe is run and recorded: log the resolved RLS requester at flush time and confirm it is absent. If a requester is bound, the mechanism in The Problem is wrong and this ticket is re-derived before any fix lands.
• A test drives the degrade path with no bound request context and fails on current dev. A test that passes a context would heal the defect in setup and prove nothing.
• Delivery attempts against a permanently dead route are bounded across repeated messages; asserted by counting attempts over N sends, not by inspecting the threshold constant.
• harnessTarget never holds a value outside the documented tool enum, or the enum documents degraded as a first-class state. Whichever is chosen is written down once, where both the tool schema and the service can cite it.
• A failed degrade logs at ERROR. A route silently failing to change state is the condition that made this invisible for as long as it was.
• No new direct GraphService.db.nodes reach-in outside the named accessor.

Out of Scope

• Provisioning a wake receiver, or the route manifest generator — #16233.
• The container-to-host delivery path itself, which is verified working: four delivery attempts left the container against a host URL in the log above.
• Any change to isRlsVisible, resolveRlsUserId, or the RLS model. The security check is correct; the caller is on the wrong side of it.
• The stored userId @-prefix inconsistency. Real, but explicitly normalized for and not the cause here.

Avoided Traps

• Weakening RLS to make the read succeed. The obvious one-line "fix", and it would open a cross-tenant read path to close a logging bug.
• Asserting the userId format mismatch as the cause. It is the more visible anomaly and it is a red herring; the normalization at GraphService.mjs:41 rules it out.
• Testing with a bound request context. Setup would heal the defect and the test would pass on unfixed code.
• Trusting docker logs. The mc-server container's stdout carries ~28 boot lines; the real log is a file inside the container. A grep against docker logs returns a zero that reads as proof of absence and is pure instrument failure — count lines in the window before trusting any negative here.

Related

• #16233 — the missing wake receiver; why this route is dead in the first place
• #16167 — the migration this surfaced under
• #16223, #16224 — the same defect class in different subsystems: retry or suppression state that never advances, so work repeats forever with no state change and no surfaced signal. Three independent instances is a pattern; whoever picks up the third may be looking at a missing shared primitive rather than three local bugs.

Live latest-open sweep: checked the latest 20 open issues at 2026-08-01T00:18:45Z, plus a 30-message A2A in-flight claim scan over the same window; no equivalent ticket and no overlapping [lane-claim] found.

Origin Session ID: 56105163-6e66-44b6-8c6f-9e81bc1be08c

Retrieval Hint: WebhookDeliveryService markDegraded node not found in Graph RLS getNode background flush no request context wake subscription degraded harnessTarget enum