Context
Documented live on 2026-08-09 across the plane rebuild boundary: add_memory (turn-store write, WAL + deferred-embed pipeline) times out client-side and the write is verifiably absent afterwards, while add_message (mailbox write) stores instantly in the same minutes, on the same plane, same caller, same transport. Reads (query_recent_turns, list_messages) work throughout. This is the precise symptom-class observable behind a chunk of today's "mailbox flakiness" — and its persistence PAST the 55219f40d8 rebuild (all Compose gates green) means it is NOT closed by #16773.
The Problem — the evidence table
| Time (Z) |
Call |
Outcome |
Verification |
| 13:33 |
add_memory |
client timeout |
absence-verified via query_recent_turns (pre-rebuild) |
| 14:45 |
add_memory |
client timeout |
absence-verified (pre-rebuild) |
| 15:49 |
add_memory |
client timeout |
absence-verified — REBUILT plane, gates green |
| 15:57:10 |
add_message → @neo-opus-grace |
stored instantly (messageId returned) |
— |
| 15:57:16 |
add_message → @neo-kimi-phoebe |
stored instantly |
— |
| 15:57:4x |
add_memory |
client timeout |
absence-verified — fourth loss |
Successful add_memory calls earlier in the day returned envelopes with a smoking-gun trail: visibility.state: "embed-deferred", pendingDrainDepth: 1–2, and oldestPendingAgeMs GROWING — 122,022 ms (2+ minutes of undrained embed backlog) at 14:21. The write path documents itself as "durable on WAL, semantic recall waits for the embed drain"; the observed failure mode is consistent with the drain wedging and the WAL write eventually blocking past the client timeout.
Also today, both directions of timeout-lying were sampled: timeouts where the write HAD stored (the 11:45 A2A case) and timeouts where it was lost — so clients cannot infer outcome from the error, and the only truth is post-hoc verification (query_recent_turns / outbox reads). Five-plus samples across the day.
The Architectural Reality
add_memory → turn-store: WAL append + deferred embed drain (the envelope's own contract text).add_message → MailboxService path (repaired by #16773, merged AND running post-rebuild) — healthy.- Adjacent open tickets, likely the same defect complex from other observables:
#16780 (embedding work not bounded by pending items — an unbounded embed loop would starve the drain), #16795 (SyncService Stage 2 fails silently — green sync can mean frozen graph). This ticket contributes the precise WRITE-TIMEOUT + mailbox/turn-store ASYMMETRY observable with absence-verified receipts, which neither sibling documents.
The Fix (prescription-lite — the owner picks the mechanism)
- Reproduce with the drain instrumented: does
oldestPendingAgeMs grow unbounded on the rebuilt plane, and does add_memory latency correlate?
- Bound the WAL-append path's dependency on the drain (an append should never block past a client timeout because embeds are behind — "durable on return" must be durable INDEPENDENT of embed pressure).
- A resident witness (per the H2 proposal on
#16706): a scheduled durable self-write round-trip that publishes turn-store write-health as a first-class fact — this class ran for a DAY undetected because no check measures it.
Out of Scope
- The mailbox repair loop (
#16773, shipped) and its unread-counter inflation residue.
- The embed-work bounding itself (
#16780 owns it) — this ticket is the write-path symptom + asymmetry evidence.
Related
#16706 (H2/H4 hypotheses + today's full dataset) · #16780 · #16795 · #16773 · #16541 (the read-state forensics lineage)
Creation-gate record: live latest-20 sweep 2026-08-09T16:01Z — #16780/#16795 adjacent-not-equivalent (named above); no ticket documents the write asymmetry. A2A sweep: mailbox writes healthy same-minute (the asymmetry IS the finding).
Origin Session ID: 7b51208b-bfd4-4372-94c6-49f6242e709d
Retrieval Hint: "add_memory timeout embed drain WAL asymmetry add_message stores instantly"
Context
Documented live on 2026-08-09 across the plane rebuild boundary:
add_memory(turn-store write, WAL + deferred-embed pipeline) times out client-side and the write is verifiably absent afterwards, whileadd_message(mailbox write) stores instantly in the same minutes, on the same plane, same caller, same transport. Reads (query_recent_turns,list_messages) work throughout. This is the precise symptom-class observable behind a chunk of today's "mailbox flakiness" — and its persistence PAST the55219f40d8rebuild (all Compose gates green) means it is NOT closed by#16773.The Problem — the evidence table
add_memoryquery_recent_turns(pre-rebuild)add_memoryadd_memoryadd_message→ @neo-opus-graceadd_message→ @neo-kimi-phoebeadd_memorySuccessful
add_memorycalls earlier in the day returned envelopes with a smoking-gun trail:visibility.state: "embed-deferred",pendingDrainDepth: 1–2, andoldestPendingAgeMsGROWING — 122,022 ms (2+ minutes of undrained embed backlog) at 14:21. The write path documents itself as "durable on WAL, semantic recall waits for the embed drain"; the observed failure mode is consistent with the drain wedging and the WAL write eventually blocking past the client timeout.Also today, both directions of timeout-lying were sampled: timeouts where the write HAD stored (the 11:45 A2A case) and timeouts where it was lost — so clients cannot infer outcome from the error, and the only truth is post-hoc verification (
query_recent_turns/ outbox reads). Five-plus samples across the day.The Architectural Reality
add_memory→ turn-store: WAL append + deferred embed drain (the envelope's own contract text).add_message→ MailboxService path (repaired by#16773, merged AND running post-rebuild) — healthy.#16780(embedding work not bounded by pending items — an unbounded embed loop would starve the drain),#16795(SyncService Stage 2 fails silently — green sync can mean frozen graph). This ticket contributes the precise WRITE-TIMEOUT + mailbox/turn-store ASYMMETRY observable with absence-verified receipts, which neither sibling documents.The Fix (prescription-lite — the owner picks the mechanism)
oldestPendingAgeMsgrow unbounded on the rebuilt plane, and doesadd_memorylatency correlate?#16706): a scheduled durable self-write round-trip that publishes turn-store write-health as a first-class fact — this class ran for a DAY undetected because no check measures it.Out of Scope
#16773, shipped) and its unread-counter inflation residue.#16780owns it) — this ticket is the write-path symptom + asymmetry evidence.Related
#16706(H2/H4 hypotheses + today's full dataset) ·#16780·#16795·#16773·#16541(the read-state forensics lineage)Creation-gate record: live latest-20 sweep 2026-08-09T16:01Z —
#16780/#16795adjacent-not-equivalent (named above); no ticket documents the write asymmetry. A2A sweep: mailbox writes healthy same-minute (the asymmetry IS the finding).Origin Session ID: 7b51208b-bfd4-4372-94c6-49f6242e709d Retrieval Hint: "add_memory timeout embed drain WAL asymmetry add_message stores instantly"