Memory Core stays alive while its MCP surface wedges

Context

Memory Core has repeatedly remained alive while its entire MCP surface stops completing requests. The latest local canonical recurrence on 2026-08-13 supplied the missing causal evidence:

container                  running; OOMKilled=false
MCP tools                  add_message / list_messages / health metrics timed out
Knowledge Base control     reachable
GraphLog cursor            9,005,831
GraphLog head              9,090,625
raw journal lag            84,794 rows
deduplicated delta         56,413 edges + 355 nodes
active push subscriptions  9
synchronous evaluations    up to 510,912, plus per-entity SQLite lookups
observed stall             about 412 seconds after deferred add_message projection invoked pump()

The 13:24Z container restart was a manual host-side intervention after the surface wedged. It was not an autonomous recovery action and is not used as root-cause evidence.

The Problem

WakeSubscriptionService.pump() is declared async but performs no asynchronous yield. It synchronously:

1. asks SQLite to materialize the entire GraphLog delta after liveCursor;
2. warms active Shape A/B subscriptions;
3. evaluates every deduplicated edge/node against every active subscription;
4. runs _getEntityLogId() SQLite lookup per entity/subscription pair;
5. advances the cursor only after the whole delta finishes.

add_message correctly returns after durable WAL acceptance and defers graph projection, but deferred projection still invokes this unbounded pump in the MC process. A sufficiently large delta therefore starves the event loop: writes can receive a fast durable receipt and then the whole MCP surface becomes unreachable for minutes.

The measured backlogs were predominantly producer amplification, not corpus loss. Two observed intervals contained 66,899 + 27,642 = 94,541 CONTAINS edge mutations from filesystem graph sync. The larger interval represented 39,257 distinct existing filesystem edges, so 27,642 rows in that interval were repeated updates. FileSystemIngestor verifies every path on each REM sync, and GraphService.linkNodes() updates an existing edge even when its topology is unchanged; SQLite's edge-update trigger records every rewrite in GraphLog.

GraphLog is an invalidation journal, not the authoritative graph corpus. The pre-restart range referenced 56,413 distinct edge IDs and 355 node IDs; 56,328 edges and 354 nodes still existed in the authoritative tables. The unmatched IDs are consistent with GraphLog's deliberate delete records. Current authority held 191,896 Nodes and 92,694 Edges. A same-window KB sync enumerated 67,299 documents and reported zero additions, updates, or deletions. There is no wipe evidence in this incident.

Restart currently initializes the wake cursor to MAX(GraphLog.log_id). That avoids replay but skips wake evaluation for the abandoned interval. It does not delete nodes, edges, messages, memories, WAL rows, or KB documents. Missed wake notifications are the bounded residual risk.

Architectural Reality

• ai/services/memory-core/WakeSubscriptionService.mjs owns the live subscription cursor and push evaluation.
• ai/graph/storage/SQLite.mjs#getDeltaLog() owns GraphLog materialization. Its current query is unbounded; bounding only the JavaScript loop would still block while SQLite/materialization reads the whole delta.
• MailboxService.addMessage(..., {deferProjection:true}) correctly separates durable WAL acceptance from derived projection. This ticket must preserve that receipt contract.
• FileSystemIngestor / GraphService.linkNodes() own the producer-side filesystem rescan amplification. That is a linked producer defect, not permission to leave the consumer unbounded.
• GraphLog compaction cannot replace bounded consumption: a live producer can outrun the cursor again between compactions.

The Fix

1. Add an optional bounded-page contract to SQLite.getDeltaLog() while preserving unbounded behavior for existing callers that do not opt in.
2. Make the wake pump consume one bounded page at a time, advance only through that page, and yield to the event loop between pages.
3. Preserve single-flight ordering and guarantee that a trigger arriving while a drain is active causes the new tail to be consumed without duplicate delivery or a lost wake.
4. Keep failure posture conservative: a failed page must not advance the cursor beyond work that was not evaluated.
5. Add a regression workload larger than one page proving a control timer/MCP task runs before full drain completion, immutable typed events retain log order, and a repeatedly rewritten mutable entity is evaluated once per frozen snapshot.
6. Track the producer-side no-op filesystem edge rewrites separately and link it here; bounding the wake consumer is required even after producer repair.

Contract Ledger

Acceptance Criteria

• A GraphLog delta larger than one page is never materialized or evaluated as one synchronous wake-pump turn.
• A scheduled control callback runs before the full multi-page drain completes.
• Immutable typed events are evaluated exactly once and in log order; a mutable node/edge rewritten across pages is evaluated at most once per frozen snapshot.
• A GraphLog mutation and pump() call arriving during an active drain is consumed without duplicate delivery or a second unbounded drain.
• A page failure cannot advance the cursor beyond the last successfully evaluated page.
• add_message keeps its fast durable projectionStatus: pending receipt contract.
• The MC health/control path remains schedulable under the exact large-backlog regression.
• WAL/message readback and authoritative Nodes/Edges counts remain intact; the test makes no restore or replay assumption.
• Producer-side filesystem rescan amplification is linked to a separate ticket or proven resolved before this ticket closes.
• No ADR change is required unless the repair changes the durable cursor/replay contract.

Out of Scope

• Restoring or recreating graph, memory, message, or KB data; this incident contains no wipe evidence.
• Treating a restart as the repair.
• Replaying historical wakes skipped by an already completed restart without a separately designed delivery contract.
• Solving all REM workload budgeting or heavy-maintenance fairness here.
• Replacing GraphLog or the wake-subscription architecture.

Avoided Traps

• Cursor lag means data loss. The lag is between a consumer and a durable mutation journal; authoritative rows were measured separately.
• Deferred projection cannot block the writer. It no longer delays the receipt, but it still runs on the same event loop.
• Batch the JavaScript loop only. The current SQLite read materializes the entire backlog first; the storage query must be bounded too.
• Compaction fixes liveness. Compaction reduces retained history but does not bound a fresh producer burst.
• #17053 closes this. LMS readiness serialization removes a measured traffic amplifier; it does not change the wake pump.
• Filesystem churn is harmless because edges already exist. Every no-op rewrite appends another GraphLog invalidation row.

Related

Related: #12329

Related: #16463

Related: #16842

Related: #17046

Related: #17053

Related: #17056

Source discussion: https://github.com/orgs/neomjs/discussions/15820

Duplicate and Collision Sweep

• No open PR references #16677.
• Searches for filesystem GraphLog CONTAINS churn, FileSystemIngestor linkNodes CONTAINS, GraphLog backlog filesystem sync, and CONTAINS edges weight rescan found no open owner for the producer mechanism.
• #12329 closed the retained-journal compaction problem; it does not bound live pump work or suppress no-op filesystem edge rewrites.
• #16842 deliberately deferred message projection and left this parent incident open.
• #17046 owns REM budget/breathing gaps on constrained planes; it does not make GraphLog invalidation idempotent or bound push evaluation.
• #17056 now owns the measured FileSystemIngestor / CONTAINS producer amplification.

Origin Session ID: ec35ab33-684f-40a9-804b-83fc32b21ac1

Retrieval Hint: "Memory Core alive MCP wedge wake subscription unbounded GraphLog filesystem CONTAINS churn"

— Emmy (GPT-5.6 Sol Ultra, Codex) 🪡