Embed-daemon: durable drain for the add_memory write-ahead log (retry/backoff)

Context

Phase 2 of the operator-settled add_memory never-fail design (#12838). Phase 1 (PR pending, Resolves #12838) ships the JSONL write-ahead log: add_memory appends the full payload durably, keeps the graph write synchronous, and defers the Chroma embed to a transitional in-process fire-and-forget (MemoryService.embedPendingMemory, mirroring the buildMiniSummary precedent). That transitional drain is correct but not durable: a process restart mid-embed leaves the record pending until the same server process happens to retry — there is no retry/backoff, no batch drain, and no cross-restart owner for the backlog.

Split per the 1-PR-per-ticket mandate (pull-request-workflow.md §9.1): #12838 closes with Phase 1; this ticket carries the Phase-2 ACs verbatim.

Live latest-open sweep: checked latest 20 open issues at 2026-06-10T14:4xZ; no equivalent daemon ticket exists (only #12838 itself names this as Phase 2).

The Problem

After Phase 1, WAL records are reconciled (embed-marked) only by the transitional in-process embed. Failure shapes left open:

1. Crash/restart mid-embed — the in-flight embed dies with the process; the record stays pending until... nothing. Recency recall still serves it via the WAL overlay, but semantic recall (query_raw_memories) never sees it.
2. Sustained embedder outage (the empirical anchor in #12838: LM Studio saturation during KB/REM backfills) — every turn's embed fails softly; the pending backlog grows with no draining owner.
3. No retry/backoff — a transient failure is retried only implicitly (never for that record).

The Architectural Reality

Phase 1 lands the complete store API the daemon needs (ai/services/memory-core/helpers/memoryWalStore.mjs):

• readPendingWalRecords({dir, ids?, limit?}) — newest-first pending backlog; corrupt-line tolerant; each record carries {id, timestamp, metadata, document, segmentKey} — exactly the collection.add triple.
• appendWalEmbedMarker({id, segmentKey}, {dir}) — reconciliation primitive (single-writer marker files; the Phase-1 transitional embed and this daemon are never concurrent writers, because step 2 below removes the transitional path in the same PR that starts the daemon).
• pruneReconciledWalSegments({dir, retentionLimit, activeSegmentKey}) — retention; never prunes a pending record.
• Purge safety is already daemon-ready: SessionService.purgeSession tombstones (embed-marks) purged-session pending records and MemoryService.embedPendingMemory carries the tenant-aware purged-session guard — the daemon reuses embedPendingMemory or replicates its guard semantics.

Config substrate: memoryWal.{dirProd,dirTest,useTestDatabase,retentionLimit,minFieldLength} leaves exist (ADR 0019 declarative leaf() + by-construction test isolation). Phase 2 adds its daemon leaves to the same block.

Placement (structural pre-flight, Stage-1 fast-path): ai/daemons/embed/daemon.mjs — sibling of ai/daemons/wake/daemon.mjs (+ kb-alerting, kb-gc, kb-reconciliation, orchestrator). NOT ai/scripts/ (PR #11008 misplacement anchor).

The Fix

1. New orchestrator-managed daemon ai/daemons/embed/daemon.mjs: poll readPendingWalRecords on a config-driven interval → embed batches via collection.add with retry/backoff + max-retries → appendWalEmbedMarker per success → pruneReconciledWalSegments per cycle.
2. Remove the transitional inline embed call from MemoryService.addMemory (keep embedPendingMemory as the shared embed primitive the daemon invokes, or hoist its guard logic into the daemon).
3. Config leaves (ADR 0019, env-var-overridable, defaults + comments in config.template.mjs): memoryWal.pollIntervalMs, memoryWal.batchSize, memoryWal.maxRetries (+ backoff base if not derived).
4. Orchestrator ownership per the #12065 SSOT pattern (task definition + supervision), mirroring how the wake daemon is managed.

Contract Ledger

Decision Record impact

aligned-with ADR 0019 (config leaves) + #12065 orchestrator-owned daemon pattern. Challenges no ADR.

Acceptance Criteria

• AC1: Orchestrator-managed daemon at ai/daemons/embed/daemon.mjs drains the JSONL WAL, embeds via collection.add with retry/backoff + max-retries, marks records done, and prunes per the retention bound. Test against a seeded JSONL (= #12838 AC4).
• AC2: After the daemon runs, the memory is semantically retrievable (query_raw_memories); the WAL record is reconciled (= #12838 AC5).
• AC3: The transitional inline embed is removed from addMemory; the WAL pending-overlay continues to keep query_recent_turns content-complete during daemon lag (existing Phase-1 specs stay green).
• AC4: Daemon config leaves (pollIntervalMs, batchSize, maxRetries) are env-var-overridable per ADR 0019; defaults in config.template.mjs with comments (completes #12838 AC6).
• AC5: The daemon honors purge semantics: tombstoned/purged-session records are never re-embedded (guard test).
• AC6 (post-merge, = #12838 AC7): Under a live heavy embed backfill, add_memory latency stays bounded and never returns MEMORY_ADD_ERROR, and the pending backlog drains once contention clears.

Out of Scope

• The write-path itself (validation gate, WAL append, sync graph write, recency overlay) — shipped in #12838 Phase 1.
• Embedding QoS/priority lanes — superseded territory (#12487/#12509 closed); the decouple makes interactive QoS moot.
• Chat-model priority lane — #12748.
• Corrupted-memory cleanup/carve-out — #12830.

Avoided Traps

• ❌ Keep the transitional in-process embed alongside the daemon — two concurrent marker-writers would break the single-writer-per-file interleave guarantee of the WAL store. The daemon PR removes the transitional path atomically.
• ❌ Place the daemon in ai/scripts/ — PR #11008 misplacement anchor; daemons live under ai/daemons/.
• ❌ In-daemon dedup via Chroma reads before each add — unnecessary: collection.add is idempotent per id at worst-duplicate level, and the marker files are the cheap source of truth; read those.

Related

• #12838 — Phase 1 (write path; the PR resolving it lands the store API + transitional embed this ticket replaces).
• #12065 — orchestrator-as-SSOT daemon pattern.
• #12123 — REM-run JSONL store precedent.

Origin Session ID: 5000ac5e-dabb-4f39-8a5c-e8ba55133f3d

Handoff Retrieval Hints:

• query_raw_memories "embed daemon WAL drain retry backoff memoryWalStore pending records"
• Code anchors: ai/services/memory-core/helpers/memoryWalStore.mjs (store API), MemoryService.embedPendingMemory (guard semantics + shared embed primitive), ai/daemons/wake/daemon.mjs (sibling pattern).

Authored by Claude Fable 5 (Claude Code), @neo-fable.