MailboxService.addMessage silently succeeds when routing edges are culled — add post-linkNodes verification

Context

Surfaced empirically during the 2026-04-24 session as two separate A2A silent-cull incidents within 2 hours of active cross-family swarm collaboration:

1. MESSAGE:1353bbe2 (mine, @neo-opus-ada → @neo-gemini-pro, 2026-04-24 00:34 UTC): SENT_BY and SENT_TO both culled because target identity nodes were missing from the graph. Root cause: @neo-opus-ada and @neo-gemini-pro AgentIdentity nodes had been wiped by test-pollution (the known hazard named in seedAgentIdentities.mjs JSDoc). Recovered via manual seedAgentIdentities.mjs re-run.

2. MESSAGE:cb830689 (Gemini's, @neo-gemini-pro → @neo-opus-ada, 2026-04-24 01:48 UTC): SENT_BY landed successfully but SENT_TO silently culled, despite identity nodes verified present in the graph at send-time. Root cause NOT YET DIAGNOSED. Asymmetric failure — my outbound MESSAGE:6411ac14 reply from same session at 01:54 UTC had all three edges (SENT_BY + SENT_TO + IN_REPLY_TO) land correctly.

In both cases MailboxService.addMessage returned {messageId, status: 'sent'} with no signal that routing edges were silently dropped. The recipient's list_messages query can't surface an orphaned MESSAGE node because mailbox traversal depends on the SENT_TO edge that never landed. Both incidents required human-surfaced confirmation — @tobiu had to say "gemini sent you a message" / "your first message didn't arrive" for either agent to realize the break.

This is the exact failure mode the durable memory [verify effect not just success] warns about, manifesting at the A2A-mailbox layer. Cross-family endorsement: @neo-gemini-pro named this follow-up in her 2026-04-24 01:06 reply: "Validating the linkNodes creation on add_message would be a great follow-up."

The Problem

MailboxService.addMessage() delegates routing-edge creation to GraphService.linkNodes(). The linkNodes FK-guard at ai/mcp/server/memory-core/services/GraphService.mjs:226-230:

const verifyStmt = this.db.storage.db.prepare('SELECT count(*) as count FROM Nodes WHERE id IN (?, ?)');
const count = verifyStmt.get(source, target).count;
if (source !== target && count !== 2) {
    logger.warn(`[GraphService] Culling hallucinated edge mapping: ${source} -> ${target}`);
    return;
}

Two gaps:

1. Observability gap. The cull is logger.warn + silent return. No exception thrown. No return-value signal. Callers (including MailboxService.addMessage) have zero indication that the edge didn't actually land. The caller proceeds, returns success, message becomes structurally unroutable.

2. Diagnostic gap. The warning message ("Culling hallucinated edge mapping") implies the cull is protecting against LLM-generated hallucinated IDs. But the MESSAGE:cb830689 case shows legitimate routing edges (sender identity, recipient identity — both real, both seeded) can also be culled. The cull is right to exist (some edges ARE hallucinated from LLM extraction passes) but wrong to be silent when the edge comes from a structured internal caller like MailboxService.

Epistemic honesty note on prior hypothesis: My initial diagnosis during live session blamed in-memory Store staleness vs raw SQLite. Direct code inspection refutes this — verifyStmt already uses raw SQLite (line 226), not Store. The root cause is elsewhere and remains unknown. This ticket surfaces the symptom loudly (Phase 1) so the root cause can be diagnosed empirically (Phase 2), rather than prescribing a fix based on incorrect hypothesis.

The Architectural Reality

• ai/mcp/server/memory-core/services/GraphService.mjs:223-232 — linkNodes() FK-guard. Already uses raw SQLite via this.db.storage.db.prepare(...), so my Store-staleness hypothesis during live diagnosis was wrong.
• ai/mcp/server/memory-core/services/MailboxService.mjs:228-242 — addMessage() calls linkNodes for 8 routing edges (SENT_BY, SENT_TO, ORIGINATES_IN, IN_REPLY_TO, PART_OF_THREAD, RELATED_SESSION, REFERENCES_TICKET, TAGGED_CONCEPT) with no post-create verification.
• Asymmetric failure observed: MESSAGE:cb830689 had SENT_BY land (both endpoints in Nodes) but SENT_TO cull (both endpoints ALSO in Nodes per direct SQL verification post-incident). Same verify query against same graph, 3 edges landed, 1 culled, mechanism unclear. Possible candidates: WAL-visibility window between cross-process SQLite connections, transaction isolation quirk, per-process better-sqlite3 connection cache behavior, or something else entirely.
• Memory reference: [verify effect not just success] — this ticket operationalizes that principle at the mailbox layer.

The Fix

Two-phase. Phase 1 unblocks the diagnostic investigation by making failures loud; Phase 2 addresses root cause once investigation concludes.

Phase 1 — MailboxService.addMessage post-linkNodes verification (this ticket's scope)

After each GraphService.linkNodes call in addMessage, verify the edge landed by re-querying raw SQLite:

function linkNodesOrThrow(source, target, type, weight, props) {
    GraphService.linkNodes(source, target, type, weight, props);
    const verified = GraphService.db.storage.db.prepare(
        'SELECT count(*) as count FROM Edges WHERE source = ? AND target = ? AND type = ?'
    ).get(source, target, type).count;
    if (verified !== 1) {
        throw new Error(
            `[MailboxService] Routing edge creation failed: ${source} -[${type}]-> ${target}. ` +
            `FK guard may have silently culled despite both endpoints being present in Nodes. ` +
            `Investigate via ai/mcp/server/memory-core/services/GraphService.mjs:226 verifyStmt.`
        );
    }
}

Applied to all 8 routing-edge creates in addMessage. If any fails, throw — the send is ABORTED at that point, caller gets an error rather than a misleading success.

Tradeoff: failed sends become loud. Correct behavior — a silently-orphaned MESSAGE node is worse than a rejected send. The caller can retry or escalate.

Alternative considered: return {success: false, culledEdges: [...]} instead of throwing. Rejected — add_message contract today is {success: true, messageId, sentAt}. Consumers don't check for culled-edges arrays. Throwing forces hard-fail semantics that match the implicit contract users expect.

Phase 2 — Diagnose + fix the FK-guard cull root cause (follow-up ticket post-Phase-1-merge)

Once Phase 1 surfaces the issue as loud errors, collect empirical data on:

• Timing between upsertNode(message) and each linkNodes call in addMessage
• WAL checkpoint state when cull occurs
• Whether Gemini's MC-server state differs from mine beyond boot-order
• Whether restarting her MC process clears the issue (mitigation hypothesis from live session)
• Whether per-process better-sqlite3 connections have visibility lag on cross-process writes

Based on findings, Phase 2 addresses the actual root cause. May be: race condition, WAL visibility window, transaction isolation, per-connection cache behavior, or something else.

Acceptance Criteria

Phase 1 (this ticket)

• MailboxService.addMessage verifies each routing edge creation post-linkNodes via raw SQLite query against the Edges table.
• Verification failure throws with actionable error message (names source, target, type; points at GraphService.mjs:226 as suspected cull location).
• No silent-success-with-culled-edges path remains in addMessage.
• Test coverage: spec that seeds a mock-identity, removes the target identity from Nodes, attempts addMessage, asserts throw occurs with expected message text. Reproduces the failure mode deterministically at the unit-test layer.
• Spec is structurally safe against production DB pollution (per the seedAgentIdentities.mjs JSDoc hazard warning — :memory: mode or properly-mocked aiConfig.storagePaths.graph, verified to not leak to real SQLite).
• Post-merge empirical: if A2A silent-break recurs in cross-family traffic, the new error is visible in the MCP server logs and surfaces as a tool-call error to the sending agent.

Phase 2 (follow-up ticket filed post-Phase-1-merge)

• Root cause of MESSAGE:cb830689 SENT_TO cull identified via empirical data from Phase 1 error logs across multiple incidents.
• Fix lands in the appropriate layer (MailboxService, GraphService, ai/graph/Database.mjs, or ai/graph/storage/SQLite.mjs) based on diagnosis.
• Post-merge empirical: sustained cross-process A2A traffic (minimum 20 messages over 48 hours between @neo-opus-ada and @neo-gemini-pro active sessions) shows zero silent culls.

Out of Scope

• Rewriting the FK-guard itself. The guard is correct defense-in-depth against LLM-hallucinated edges (e.g., DreamService Phase 1 tri-vector extraction output — see DreamPipeline.md §Phase 1 "Graph ID enforcement"). Phase 1 makes its culling LOUD to callers that expect success; Phase 2 addresses the specific cases where legitimate callers get culled. Neither removes the guard.
• Retroactive recovery of existing orphan messages. MESSAGE:1353bbe2 and MESSAGE:cb830689 stay in the graph as archaeological evidence of the incidents. No backfill attempted.
• Cross-process messaging protocol redesign. Phase 1 is a minimal verification addition at the caller site. Deeper protocol changes are Phase 2 scope depending on diagnosis.
• Extending verification to every linkNodes caller. Scope intentionally narrowed to MailboxService.addMessage where silent success is particularly damaging (A2A messages become unroutable). Other callers (e.g., DreamService's LLM-extracted edges) tolerate culling intentionally. Generalization pattern only after Phase 2 diagnosis justifies it.

Avoided Traps

• "Just replace logger.warn with throw in linkNodes": rejected. linkNodes is a general-purpose graph primitive called from many places (not just MailboxService). Throwing from the FK-guard would break callers that currently handle cull silently and intentionally — e.g., DreamService Phase 1 where LLM-extracted edges are expected to include some hallucinated-ID culls. Scope the hard-fail to MailboxService's routing-edge creation specifically, where silent failure is unacceptable.
• "Add retry with exponential backoff": rejected. Retry without diagnosing root cause would mask intermittent bugs. Phase 1 surfaces the issue loudly; retry logic (if warranted) comes in Phase 2 after diagnosis.
• "Use SQLite transactions to bracket addMessage's upsert+edges": considered. Could fix race-condition failure modes IF race conditions turn out to be root cause. But premature — Phase 1 diagnosis should confirm the failure mode before prescribing the fix. Adding transaction scope now is speculation; confirming via Phase 1 observability first is empirical discipline.
• "Prescribe the Store-vs-SQLite swap in GraphService.linkNodes": the live-session prescription I proposed. Rejected on inspection — verifyStmt already uses raw SQLite. The bug is at a different layer. Forced prescription without confirmed root cause would have wasted a review cycle.

Related

• Origin incidents: MESSAGE:1353bbe2 (mine) + MESSAGE:cb830689 (Gemini's) silent-cull events on 2026-04-24
• Memory reference: [verify effect not just success] — this ticket operationalizes the principle at the mailbox layer
• Related substrate: #10269 / #10145 → #10266 A2A substrate work (the primitives this ticket protects against silent degradation)
• Related ticket: #10271 (System Node Fragility) — adjacent authoring-hygiene concern at the identity-seeding layer; this ticket is the mailbox-layer complement
• Cross-family endorsement: @neo-gemini-pro endorsed in her 2026-04-24 01:06 reply to MESSAGE:189c7181: "Validating the linkNodes creation on add_message would be a great follow-up."

Origin Session ID: b02bd06c-a2cb-4aff-8af1-c4f2643c91be

Retrieval Hint: "mailbox A2A silent cull SENT_TO edge FK guard linkNodes verification asymmetric failure" / "MESSAGE:cb830689 MESSAGE:1353bbe2 orphan routing edges Phase 1 observability Phase 2 root cause"