Context
Live get_rem_pipeline_state this session: undigested 316 (~stable from ~326 = not draining) + recentCycles: [] (no REM cycle recorded) while the Golden Path handoff regenerated fresh (golden-path decoupled by #13783). The orchestrator is alive but the consolidation half is silently stalled — the operator's "orchestrator does still not fully work." #13551 (embedDrainLivenessWatchdog) watches the embed-drain WAL backlog exactly this way (in the recovery incident the embed-drain died silently and no alarm fired for ~8 days); nothing watches the REM / consolidation cycle. ADR 0023 AC-3 (consolidation-liveness: observable, never assumed-green) mandates closing this gap.
The Problem
A stalled REM cycle produces no alarm: the forecast keeps regenerating (decoupled) while the graph rots — the "green-but-rotting" anti-pattern (ADR 0023 §5). recentCycles: [] is invisible until a human notices stale routing. Process-alive ≠ consolidation-draining (the same lesson #13551 encodes for embed-drain).
The Architectural Reality
Mirror ai/daemons/orchestrator/scheduling/embedDrainLivenessWatchdog.mjs (#13551) — read-only, never-fail, one-shot-latch, pure getDueTask:
- Read primitive:
ai/services/memory-core/helpers/remRunStateStore.mjs — the last successful REM-cycle timestamp (the recentCycles source) + the undigested backlog count (get_rem_pipeline_state axes).
- Stall signal (mirrors embed-drain's
oldestAgeMs > thresholdMs): time-since-last-successful-REM-cycle > threshold AND undigested backlog > 0. Fail-soft (a read fault → no alarm, never a false stall, never a throw into the pipeline).
- Register in
registry.mjs as a read-only health-check with backpressure: 'none' + empty dependencies (the #13551 precedent — never takes a heavy lease).
- Config leaves (
remConsolidationWatchdogCheckMs, staleness threshold) per ADR 0019.
- One-shot latch: alarm on stall-onset, latched, cleared on a healthy check.
Decision Record impact
aligned-with ADR 0023 (AC-3 consolidation-liveness — this is the observability execution). Sub of #13624 (orchestrator heavy-maintenance backlog doesn't drain — the observability angle; "peers pick up the angles").
Acceptance Criteria
Out of Scope
- FIXING the stall (the lease starvation → #13624 / #13780; over-band digestion → #12439). This is observability only — read-only alarm.
- The handoff-visibility surface (#13807, grace's — surfaces the gap in
sandman_handoff.md); this watchdog is the complementary active scheduled alarm.
Related
Parent: #13624. Pattern: #13551 (embedDrainLivenessWatchdog). Complementary: #13807 (handoff-visibility). Mandated-by: ADR 0023 (#13805, AC-3). Adjacent: #13780 (lease-yield), #12439 (over-band).
Release classification: boardless (observability hardening; not release-blocking).
Origin Session ID: bd479771-fba8-405a-8d0c-be86a73ee8fc
Retrieval Hint: query_raw_memories("REM consolidation-liveness watchdog recentCycles stalled dream cycle alarm #13551 pattern #13624")
Authored by @neo-opus-vega (Vega).
Context
Live
get_rem_pipeline_statethis session: undigested 316 (~stable from ~326 = not draining) +recentCycles: [](no REM cycle recorded) while the Golden Path handoff regenerated fresh (golden-path decoupled by #13783). The orchestrator is alive but the consolidation half is silently stalled — the operator's "orchestrator does still not fully work." #13551 (embedDrainLivenessWatchdog) watches the embed-drain WAL backlog exactly this way (in the recovery incident the embed-drain died silently and no alarm fired for ~8 days); nothing watches the REM / consolidation cycle. ADR 0023 AC-3 (consolidation-liveness: observable, never assumed-green) mandates closing this gap.The Problem
A stalled REM cycle produces no alarm: the forecast keeps regenerating (decoupled) while the graph rots — the "green-but-rotting" anti-pattern (ADR 0023 §5).
recentCycles: []is invisible until a human notices stale routing. Process-alive ≠ consolidation-draining (the same lesson #13551 encodes for embed-drain).The Architectural Reality
Mirror
ai/daemons/orchestrator/scheduling/embedDrainLivenessWatchdog.mjs(#13551) — read-only, never-fail, one-shot-latch, puregetDueTask:ai/services/memory-core/helpers/remRunStateStore.mjs— the last successful REM-cycle timestamp (therecentCyclessource) + the undigested backlog count (get_rem_pipeline_stateaxes).oldestAgeMs > thresholdMs): time-since-last-successful-REM-cycle > threshold AND undigested backlog > 0. Fail-soft (a read fault → no alarm, never a false stall, never a throw into the pipeline).registry.mjsas a read-onlyhealth-checkwithbackpressure: 'none'+ emptydependencies(the #13551 precedent — never takes a heavy lease).remConsolidationWatchdogCheckMs, staleness threshold) per ADR 0019.Decision Record impact
aligned-withADR 0023 (AC-3 consolidation-liveness — this is the observability execution). Sub of #13624 (orchestrator heavy-maintenance backlog doesn't drain — the observability angle; "peers pick up the angles").Acceptance Criteria
ai/daemons/orchestrator/scheduling/remConsolidationLivenessWatchdog.mjs— mirrors #13551 (getRemCycleStalenessread-only/fail-soft +evaluateStallAlarmone-shot latch + puregetDueTask).registry.mjsas a read-onlyhealth-check,backpressure: 'none',dependencies: [].<= 0disables the lane.health-checkexecute branch records the outcome — pattern parity with #13551).Out of Scope
sandman_handoff.md); this watchdog is the complementary active scheduled alarm.Related
Parent: #13624. Pattern: #13551 (
embedDrainLivenessWatchdog). Complementary: #13807 (handoff-visibility). Mandated-by: ADR 0023 (#13805, AC-3). Adjacent: #13780 (lease-yield), #12439 (over-band).Release classification: boardless (observability hardening; not release-blocking).
Origin Session ID: bd479771-fba8-405a-8d0c-be86a73ee8fc Retrieval Hint: query_raw_memories("REM consolidation-liveness watchdog recentCycles stalled dream cycle alarm #13551 pattern #13624")
Authored by @neo-opus-vega (Vega).