Embed-drain liveness watchdog: alarm on a silently-stalled WAL backlog

Context

Recurrence-guard follow-up from the #13495 embed-drain recovery (the silent-drain-death incident, 2026-06-19). The recovery repaired the immediate failure — the single sole-drainer (#12864) stopped embedding WAL records, ~2,500 add_memory records orphaned un-embedded across 6 clones, semantic recall frozen since ~06-10 — via the shared-WAL-dir topology (#13542/#13545), the per-append write-lock (#13543 / #13544, MERGED), and a one-shot backlog recovery (Part 3).

But the recovery fixes the backlog and the topology; it does NOT close the detection gap: the drain died and no alarm fired for ~8 days. The orchestrator supervises the embed daemon's process existence, but nothing monitors drain progress — so a dead/stalled drain (or an un-embedded-backlog buildup from any cause) is invisible until a human notices stale recall.

Client-relevant: a silently-stalled embed pipeline in a cloud Agent OS deployment is a serious, hard-to-notice reliability failure (operator, 2026-06-19: "agent OS issues will affect Client too").

The Problem

The embed pipeline is fire-and-forget by design — add_memory is the never-fail turn-save (appendWalMemory is durable; embed is decoupled). That decoupling is correct, but it means a dead embedder produces no user-visible error; the WAL just grows un-reconciled. The only signal today is a human noticing degraded semantic search days later. We need a liveness watchdog: a periodic, read-only check of embed-drain progress that raises an alarm when the un-embedded backlog is stale beyond a threshold.

The Architectural Reality

• The pending-backlog primitive already exists: ai/services/memory-core/helpers/memoryWalStore.mjs — readPendingWalRecords (appended-but-not-yet-embed-marked) + getMissingMemoryWalLeaves + the UTC-day WAL segment keys give both count and age of the un-embedded backlog.
• The drain is gated by the sole-drainer lock (ai/daemons/embed/drainLock.mjs, #12864); the embed daemon is spawned + supervised by the orchestrator (ai/daemons/orchestrator/), conditional on embedDaemonEnabled.
• The orchestrator already runs a supervision loop — the natural host for a progress-liveness check — or this could be a scheduled health-check task (ADR 0014 scheduler taxonomy). Substrate convergence is part of this ticket.

The Fix

A read-only watchdog that periodically computes the age of the oldest un-embedded WAL record (and/or pending count) and alarms when it exceeds a threshold (e.g. oldest-pending > 1h on the drainer clone → the drain has stalled):

• Substrate (converge in design): primary candidate = the orchestrator supervision loop (it already owns embed-daemon lifecycle); alternative = a scheduled health-check task. MUST be read-only (never touch the never-fail add_memory write path).
• Signal: an alarm A2A to the swarm/operator (and/or a health-metric surface) when the threshold trips, carrying the stalled-since timestamp + pending count.
• Threshold: tunable; conservative default (hours, not days) so a dead drain surfaces same-session, not after a week.

Contract Ledger Matrix

Decision Record impact

aligned-with #12864 (sole-drainer lock) + the #13495 recovery; if implemented as a scheduled task, aligned-with ADR 0014 (cloud scheduler-task taxonomy). No ADR challenged.

Acceptance Criteria

• A periodic, read-only liveness check computes the age (and/or count) of the oldest un-embedded WAL record.
• When the oldest un-embedded record exceeds a tunable threshold, an alarm is raised (A2A to swarm/operator and/or a health-metric surface) carrying the stalled-since timestamp + pending count.
• The watchdog is strictly read-only — it never touches or blocks the never-fail add_memory / appendWalMemory write path; a watchdog failure degrades to "no alarm," never to a write failure.
• The substrate decision (orchestrator supervision loop vs scheduled health-check task) is documented with rationale.
• Unit coverage: threshold-trip + no-trip paths over WAL fixtures with an injected clock (no real drain/daemon spawned).

Out of Scope

• The #13495 backlog recovery itself (the one-shot ~2,500-record reconcile — Grace/@tobiu).
• The shared-WAL-dir topology (#13542/#13545) + the per-append lock (#13544) — already delivered.
• Embedding-model instance consolidation (#13539 — separate resource concern).
• Auto-restarting a dead drain — this ticket DETECTS + ALARMS; auto-remediation is a separable follow-up.

Avoided Traps

• Putting the check on the write path. add_memory is never-fail by design; the watchdog must observe the WAL read-only, never gate or slow the durable append.
• Process-existence ≠ progress. The orchestrator already checks the daemon process exists; that did NOT catch the silent death. The watchdog must measure progress (backlog age), not just process liveness.

Related

• Recovery: #13495 (parent recovery), #13542 / #13545 (Part 1), #13543 / #13544 (Part 2 — the lock, MERGED), #12864 (sole-drainer drain-lock).
• Sibling embed-pipeline-health: #13539 (qwen embedding-instance consolidation), #13496 (Chroma stored-embedding export repair), #13503 (Chroma vector-index coverage audit).

Release classification: Agent OS reliability hardening (post-recovery recurrence-guard) — boardless (not v13-release-blocking).

Origin Session ID: d1903453-9b2d-4f5d-bb5a-cf51ac2f7538

Retrieval Hint: query_raw_memories "embed-drain liveness watchdog silent drain death WAL backlog alarm"; the #13495 recovery incident (2026-06-19); memoryWalStore.readPendingWalRecords.

Authored by Claude Opus 4.8 (Claude Code), @neo-opus-vega (Vega).