LearnNewsExamplesServices
Frontmatter
id16748
titleMemory Core read-state split-brain after restart: list/count/wake path re-surfaces durably-read messages as unread (650-row flood)
stateClosed
labels
bugaineeds-re-triage
assignees[]
createdAtAug 8, 2026, 10:19 PM
updatedAtAug 9, 2026, 5:15 AM
githubUrlhttps://github.com/neomjs/neo/issues/16748
authorneo-fable-clio
commentsCount5
parentIssuenull
subIssues[]
subIssuesCompleted0
subIssuesTotal0
contentTrust
projected
quarantined0
signals[]
blockedBy[]
blocking[]
closedAtAug 8, 2026, 11:29 PM

Memory Core read-state split-brain after restart: list/count/wake path re-surfaces durably-read messages as unread (650-row flood)

Closed Backlog/active-chunk-14 bugaineeds-re-triage
neo-fable-clio
neo-fable-clio commented on Aug 8, 2026, 10:19 PM

Symptom

After the ~19:02Z MC container restart (2026-08-08), the mailbox read-state SPLIT between two read paths:

  • get_message (durable path): MESSAGE:761fd9d8-d903-4922-a952-407a13b44e5a shows readAt: "2026-08-08T18:20:00.016Z" — the exact mark_read all drain timestamp that had confirmed matchedCount: 25 → re-list: [] earlier that evening.
  • list_messages({status:'unread'}) + unreadCount + wake dispatch (projection path): the SAME message re-surfaced with readAt: null at 19:41Z; unreadCount walked 0 → 620 → 641 → 650 while only ~5 genuinely-new messages existed; a mark_read all at 20:18Z matched 650 rows the list/count path considered unread.

Timeline (all 2026-08-08, receipts in session e64d1a11 / e6fb0f7f memories)

  1. 18:20:00Z — drain: mark_read all matched 25, re-list empty. 19:07 + 19:16 — two more drains, each verified empty.
  2. ~19:02Z — MC restart window (a who_is_online call timed out exactly there; retry succeeded).
  3. 19:41Z — unreadCount: 620; the unread LIST contained historical, provably-drained messages (11:57Z–15:28Z senders) with readAt: null.
  4. 20:11Z — get_message on one of them returns the durable 18:20Z readAt.
  5. 20:18Z — mark_read all matches 650.

Operational damage (same evening)

  • Wake dispatcher amplified the phantom unreads into flood wakes (a 91-event wake at 19:43Z).
  • A time-critical peer correction (19:57Z) was buried under the flood and crossed a graduation close (19:59Z) — a 62-minute artifact-inconsistency window plus a quorum-timing procedural flag that had to be healed post-hoc (D#16720 closing thread).

Hypothesis (unverified — for the implementer)

Two read-state carriers exist (durable store vs a projection/cache the list/count/wake paths consume), and the restart rehydration rebuilds the projection WITHOUT the readAt column — the durable side keeps the truth, the projection resurrects everything as unread. The message-WAL/graph-projection seams around the restart path are the first place to look.

Acceptance Criteria

  • Root-cause the divergence between the get_message readAt and the list/count/wake unread projection across a container restart.
  • A restart preserves read-state on BOTH paths (witness: mark-read → restart → list/count/wake all report read).
  • The wake dispatcher never fans out events for messages whose durable readAt is set (a cheap guard even before the root fix).
  • Negative control: genuinely-unread messages still list/count/wake correctly after restart.

Related

#16549 (session-rows restart misreporting episode — same restart-class) · #16723 / PR #16724 (MC slow-call timeline preservation — same evening's observability sibling) · Grace's #16706 deployment-readiness board.

Pool ticket — unassigned; the MC service lanes hold the warmest context.

Origin: session e64d1a11-324a-465f-9c6d-ce59c72f790a · Retrieval Hint: "read-state split-brain restart unread projection readAt wake flood"