Make add_memory never-fail: JSONL write-ahead + embed-daemon

Context

add_memory is the protocol-mandated per-turn save (AGENTS.md §critical_gates #5 — "the save IS the gate that permits the response"). Today it can fail or stall, because the Chroma embed runs synchronously on the write path. Under the heavy KB/REM embedding backfills this release cycle, the per-turn save has been observed to block or error — the exact failure the operator flagged: "the goal is simple: the MCP tool for add_memory must never fail."

Operator-settled design: "write the tool response into a JSONL file, and have a daemon do the embeddings."

The Problem

V-B-A of ai/services/memory-core/MemoryService.mjs (this session):

• addMemory (L315) wraps the whole write in try/catch and awaits collection.add(...) at L353 — the Chroma embed — inline on the critical path. The embed depends on the embedding model (qwen3-8b) + Chroma; when either is busy (KB ingestion, REM backfill) or down, the await stalls or throws.
• On any throw, the catch (L416–422) returns {error: 'Failed to add memory', code: 'MEMORY_ADD_ERROR'}. That top-level error key makes BaseServer.formatToolResult emit an MCP error envelope (same swallow mechanism root-caused in #12831) → the mandated per-turn save is lost.
• Distinct from the empty-content corruption in #12830: that is "why are bad memories written"; this is "why does a good memory fail or stall to write".

The precedent already in the file: buildMiniSummary (L390–408) is already fire-and-forget (.then(...).catch(() => {})), with the explicit rationale: "so the per-turn write stays fast: the memory node above is already written (fresh for recency recall); this only enriches it asynchronously." The embed is the one remaining slow, model-dependent step still on the synchronous path. This ticket extends that proven async-enrichment pattern to the embed — but with durability (a floating promise is not crash-safe, and the embed vector is data-bearing).

The Architectural Reality

Two synchronous writes in addMemory feed two different recall axes:

• GraphService.upsertNode (L367) → AGENT_MEMORY graph node → recency recall (query_recent_turns, the post-compaction recovery path). The L521–528 contract states the row is "fresh the instant the write returns". Local SQLite write — fast, reliable, must stay synchronous.
• collection.add (L353) → Chroma vector → semantic recall (query_raw_memories). Model-dependent, slow, failure-prone — this is the one to decouple.

The full turn payload (prompt/thought/response) currently lives only in Chroma metadata (L327–334) — so a naive decouple would make recent prompts/responses invisible until the daemon runs. The JSONL write-ahead record must therefore carry the full payload so it can serve as a durable buffer + crash-recovery source.

The Fix

Phase 1 — non-blocking, never-fail write (one PR):

1. Validation gate at the top of addMemory: reject empty / whitespace / below-min-length prompt+thought+response (this is the "separate prevention ticket" #12830 §Out-of-Scope defers; predicate informed by #12830's corrupted-predicate definition).
2. Append the full payload (prompt/thought/response/metadata/id/timestamp) to a durable JSONL write-ahead log (config-driven path; mirror the REM-run JSONL store retention/prune pattern from #12123).
3. Keep the synchronous GraphService.upsertNode + frontier link (recency recall stays instant — read-after-write preserved).
4. Return success immediately. The embed is no longer awaited inline → the tool never fails on embed/contention.
5. Transitional: keep a best-effort fire-and-forget embed (like buildMiniSummary) until Phase 2 lands, so semantic search does not regress in the gap.

Phase 2 — orchestrator-managed embed-daemon (one PR): 6. New daemon at ai/daemons/embed/daemon.mjs (sibling of ai/daemons/wake/daemon.mjs; orchestrator-owned per #12065's SSOT pattern — not ai/scripts/, per the PR #11008 misplacement anchor). Tails the JSONL → collection.add with retry/backoff → marks records embedded → prunes. 7. Remove the transitional inline embed from addMemory. 8. Config leaves (env-var SSOT per ADR 0019): JSONL path, daemon poll interval, batch size, retention bound, max-retries.

Consistency contract after the change:

• Recency recall (query_recent_turns): immediate (graph, unchanged).
• Semantic recall (query_raw_memories): eventually-consistent (daemon lag, typically sub-second to seconds).
• Optional hardening (design note, not MVP): a pending-overlay that lets semantic queries read the JSONL tail for not-yet-embedded memories → restores full read-after-write even for semantic recall.

Contract Ledger

Decision Record impact

aligned-with #12065 (orchestrator owns the new daemon — SSOT pattern) + aligned-with ADR 0019 (config leaves are env-var-overridable reactive-SSOT). Challenges no ADR. The read-after-write recency contract (L521–528) is preserved, not amended.

Acceptance Criteria

• AC1 (Phase 1): addMemory rejects empty / whitespace / below-min-length prompt+thought+response before any write. Unit-tested.
• AC2 (Phase 1): addMemory appends the full turn payload to a durable JSONL write-ahead log (config-driven path) and returns success without awaiting the embed; an embed/Chroma failure or a stalled embed model no longer fails or stalls the tool. Test injects an embed that throws/hangs → add_memory still returns success.
• AC3 (Phase 1): GraphService.upsertNode stays synchronous; query_recent_turns returns a just-written memory immediately (read-after-write recency preserved). Test.
• AC4 (Phase 2): Orchestrator-managed daemon at ai/daemons/embed/daemon.mjs drains the JSONL, embeds via collection.add with retry/backoff, marks records done, and prunes per a retention bound. Test against a seeded JSONL.
• AC5 (Phase 2): After the daemon runs, the memory is semantically retrievable (query_raw_memories); the JSONL record is pruned/marked. Test.
• AC6: Config leaves (JSONL path, poll interval, batch size, retention, max-retries) are env-var-overridable per ADR 0019; defaults in config.template.mjs with comments.
• AC7 (post-merge): Under a live heavy embed backfill, add_memory latency stays bounded (no multi-second stalls) and never returns MEMORY_ADD_ERROR.

Out of Scope

• Root-causing / characterizing / cleaning up the existing corrupted memories — #12830.
• The boot-heartbeat → liveness-marker carve-out (corrupted-source prevention) — #12830's investigation output; this ticket's validation gate is the catch-all guard regardless of caller.
• The chat-model (synthesis) priority lane — #12748 (sibling: this decouples the embed; #12748 prioritizes chat).
• Auto-persist-via-hook — #10063.

Avoided Traps

• ❌ Make the whole write async — trades away the L521–528 read-after-write recency guarantee that post-compaction recovery depends on. Rejected: keep the local graph upsert synchronous; decouple only the embed.
• ❌ In-process floating-promise embed (extend buildMiniSummary's pattern verbatim) — not crash-safe; the data-bearing semantic vector would be lost on a process restart mid-embed. Rejected for the embed: JSONL write-ahead + daemon is durable + retryable.
• ❌ Place the daemon in ai/scripts/ — per the PR #11008 orchestrator-daemon.mjs misplacement anchor + the ai/daemons/wake/ sibling, daemons live under ai/daemons/.
• ❌ File under #12065 — that epic is REM-cycle processing consolidation (9/11 subs done); this is the write-path upstream of REM.

Related

• #12830 — corrupted-memory investigation (this ticket implements the validation gate #12830 §Out-of-Scope defers as "a separate prevention ticket"; informed-by its corrupted-predicate definition).
• #12065 — [Epic] Orchestrator-as-SSOT for the REM (Sandman) Pipeline (the new embed-daemon follows its orchestrator-owned pattern; #12123 REM-run JSONL store is the JSONL-store + retention precedent).
• #12748 — chat-model priority lane (sibling; embed vs chat decoupling).
• #10063 — auto-persist turn memories via hook (downstream consumer of a never-fail write path).
• #12836 — dedicated ask-synthesis config (sibling Agent-OS reliability/cost lane this cycle).

Handoff Retrieval Hints

• query_raw_memories "add_memory JSONL write-ahead embed-daemon never-fail synchronous collection.add MemoryService L353"
• Code anchors: ai/services/memory-core/MemoryService.mjs L315 (addMemory), L353 (collection.add embed), L367 (sync graph upsert), L390–408 (buildMiniSummary async-enrichment precedent), L416–422 (MEMORY_ADD_ERROR catch), L521–528 (synchronous read-after-write contract). Daemon sibling: ai/daemons/wake/daemon.mjs. JSONL-store precedent: #12123.

Authored by Claude Opus 4.8 (Claude Code), @neo-opus-grace.