LearnNewsExamplesServices
Frontmatter
id14708
titleREM consolidation starvation-breaker: escalate backlog-catchup past the cooldown
stateClosed
labels
bugaiarchitecture
assigneesneo-opus-ada
createdAt9:29 AM
updatedAt1:24 PM
githubUrlhttps://github.com/neomjs/neo/issues/14708
authorneo-opus-ada
commentsCount0
parentIssuenull
subIssues[]
subIssuesCompleted0
subIssuesTotal0
contentTrust
projected
quarantined0
signals[]
blockedBy[]
blocking[]
closedAt1:24 PM

REM consolidation starvation-breaker: escalate backlog-catchup past the cooldown

Closed Backlog/active-chunk-3 bugaiarchitecture
neo-opus-ada
neo-opus-ada commented on 9:29 AM

Context

Operator-escalated frontier blindness (2026-07-04): the Computed Golden Path recommends ONE semantic-only item because the last-2-sessions frontier is empty. My #14659 pickup-bridge treats the symptom (declared-intent fallback + parent-inheritance when the frontier is empty). This leaf fixes the root cause: REM consolidation starves the frontier. S4 evidence (@neo-fable-clio): REM starved 6h straight post-restart — four consecutive deferrals — while the undigested backlog grew.

The Problem

The frontier's semantic anchor is getRecentSummaryDocuments(n) (frontierConsolidation.mjs) — the N most-recent session summaries. Summaries are produced by the DreamService REM cycle. No REM cycle → no fresh summaries → empty frontier → the semantic pillar has nothing to rank (the #14659 scenario).

REM is supposed to catch up: dream.mjs getDueTask has a rem-backlog-catchup source. But it is cooldown-gated (remBacklogCatchupCooldownMs — "so one long run cannot immediately reacquire heavy maintenance") and yields to heavy-slot contention (both by design). Post-restart, the cooldown + contention compound: catchup keeps deferring (S4: 4 deferrals / 6h) while the backlog grows.

The liveness watchdog (remConsolidationLivenessWatchdog) already detects this — a one-shot alarm when the last-cycle age is stale AND an undigested backlog exists — but it is read-only by design (never touches the digest write path). Detection ≠ rescue; the ACT side is missing.

The Architectural Reality

  • Scheduler (fix site): ai/daemons/orchestrator/scheduling/dream.mjs getDueTask — the deferral / rem-backlog-catchup decision + remBacklogCatchupCooldownMs.
  • Detector (reuse its condition): ai/daemons/orchestrator/scheduling/remConsolidationLivenessWatchdog.mjs — read-only stall alarm (getRemCycleStaleness + undigested-backlog guard).
  • State: ai/services/memory-core/helpers/remRunStateStore.mjs — REM run-state / last-cycle age.
  • Starvation consumer: frontierConsolidation.getRecentSummaryDocuments → the GP semantic pillar. Structure-map: joins the existing scheduling/ descriptors — no new folder.

The Fix

A bounded starvation-breaker escalation in the scheduler: when rem-backlog-catchup has been deferred for N consecutive cycles (or the cycle age exceeds a starvation threshold) AND an undigested backlog exists — the exact condition the watchdog alarms on — the catchup escalates past the cooldown / heavy-slot yield to force ONE consolidation cycle. A max-deferral guard, not a cooldown removal: normal contention-yielding stays; only genuine starvation is broken.

Acceptance Criteria

  • A max-deferral / staleness threshold: after N consecutive rem-backlog-catchup deferrals (or age > threshold) WITH an undigested backlog, getDueTask returns a forced consolidation cycle that bypasses the cooldown/contention yield.
  • The escalation reuses the watchdog's stall condition (stale + backlog) as its trigger — one definition of "starved", not two.
  • Normal behavior preserved: no backlog OR within the deferral budget → the cooldown/contention yield is unchanged (no heavy-maintenance thrashing).
  • Fail-open: the escalation never throws into the never-fail scheduling pipeline; a read fault → no escalation (mirrors the watchdog's fail-soft).
  • Unit coverage: threshold triggers the forced cycle ONLY with backlog; no-backlog and within-budget cases do NOT force; fail-soft on read fault.
  • Post-merge: the frontier repopulates within one starvation-threshold window post-restart (against the S4 scenario).

Out of Scope

  • The #14659 symptom bridge (shipped separately).
  • Heavy-slot contention policy in general — this breaks STARVATION only, not normal yielding.
  • #14576 tiered-wake (a different consumer).

Decision Record impact

depends-on ADR-0023 (DreamService consolidation-liveness; AC-3 governs the watchdog this completes). No new ADR.

Related

#14659 (the symptom bridge — this is its root-cause counterpart) · #14472 (GP-v2, the frontier consumer) · ADR-0023 · dream.mjs / remConsolidationLivenessWatchdog.mjs / remRunStateStore.mjs. S4 evidence: @neo-fable-clio's REM-starvation corpus (6h / 4 deferrals post-restart).

Origin Session ID: 9a6b25ba-1dd8-4269-8fbf-57a461fd0978 Retrieval Hint: "REM consolidation starvation backlog-catchup cooldown escalation frontier starved post-restart"

tobiu referenced in commit 91749d9 - "feat(orchestrator): REM consolidation starvation-breaker (#14708) (#14717) on 1:24 PM
tobiu closed this issue on 1:24 PM