Context
During local validation of the REM graph-extraction backlog, the orchestrator showed completion state behaving correctly but backlog drain behaving too slowly. At 2026-06-24T21:14:39Z, the latest 20 open issue sweep found no equivalent open ticket; the recent A2A claim sweep found no overlapping claim; local resources/content search found prior discussion analysis of the 10-per-hour cap but no actionable open child ticket.
The key observation: the 21:01 session summarization run completed and released its heavy-maintenance state at 21:08:11, so this is not an indefinitely stale running flag. The remaining defect is that REM goes back to the normal hourly cadence even when the previous REM pass saturated the configured work batch.
Release classification: boardless follow-up under #13755; not release-board scoped unless an operator explicitly retargets it.
The Problem
REM consolidation currently treats a large undigested-session backlog like normal steady-state maintenance. Default cadence is hourly, while the REM batch limit is 10 sessions per cycle. A current local REM run-state file showed sessionsProcessed: 10, exactly saturating the configured batch. With a backlog around 286 missing REM extractions, that implies roughly 29 successful hourly cycles before deferrals or competing heavy-maintenance lanes stretch it further.
This is too slow for a catch-up backlog. The system already knows enough to infer that more work probably remains: the completed REM cycle processed the maximum configured batch. That signal is recorded in REM run-state and health payloads, but it is not fed back into the scheduler due calculation.
The Architectural Reality
- ai/config.mjs:503 defines AiConfig.orchestrator.intervals.dreamMs as the normal hourly REM cadence.
- ai/mcp/server/memory-core/config.mjs:145 defines remSleepBatchLimit as 10 sessions per REM cycle.
- ai/daemons/orchestrator/scheduling/dream.mjs:54 only gates the next REM run by elapsed time since the cadence anchor.
- ai/daemons/orchestrator/scheduling/pipeline.mjs:477 builds a recordPayload with sessionsProcessed, but pipeline.mjs:487 calls taskStateService.markCompleted(taskName) without persisting that work signal into task state.
- ai/daemons/orchestrator/services/TaskStateService.mjs:190 only records success timing and exit code on completion.
The result is a missing feedback edge: REM reports work saturation, but the scheduler cannot see it cheaply on the next poll.
The Fix
Add backlog-aware REM catch-up scheduling without lowering the normal hourly cadence:
- Persist REM completion metadata on the dream task state, at minimum sessionsProcessed and whether the completed batch saturated remSleepBatchLimit.
- Extend the pure dream scheduler so a successful, non-overflowed, saturated REM cycle becomes due again after a short catch-up cooldown.
- Return to normal hourly cadence when the previous cycle was skipped, failed, non-saturated, or overflowed.
- Keep the heavy-maintenance lease invariant: no parallel REM cycles and no competing heavy task overlap.
- Keep defaults working out of the box; no deployment env should be required.
Contract Ledger Matrix
| Target Surface |
Source of Authority |
Proposed Behavior |
Fallback |
Docs |
Evidence |
| Dream task state |
TaskStateService + orchestrator task state JSON |
Persist bounded REM outcome metadata needed by scheduler catch-up |
Existing timing-only state remains readable |
JSDoc on new state helper / fields |
Unit test asserts metadata persistence |
| Dream scheduler |
ai/daemons/orchestrator/scheduling/dream.mjs |
Schedule short-cooldown catch-up only after saturated successful non-overflow REM cycles |
Normal dreamMs cadence |
JSDoc on due-trigger policy |
Unit tests for saturated, non-saturated, skipped/failed/overflow paths |
| REM service outcome |
DreamService.executeRemCycle() outcome envelope |
Continue emitting sessionsProcessed; caller derives saturation against config-owned batch limit |
Existing outcome shape stays compatible |
Existing run-state docs/comments |
Unit or focused service test uses synthetic outcome |
Decision Record impact
Aligned with ADR 0019: config values must stay in AiConfig / existing Memory Core config leaves; no hidden defaults, no env re-derivation, and no config pass-through across modules beyond local leaf reads.
Acceptance Criteria
Out of Scope
- Changing REM extraction semantics, chunking, tokenizer guards, or graph node schema.
- Increasing local model parallelism or changing provider model selection.
- Lowering the normal hourly REM cadence globally.
- Solving adaptive downscaling / hardware sweet-spot tuning from discussion #13873.
Avoided Traps
- Do not simply raise remSleepBatchLimit: that makes each heavy lease larger and can recreate long blocking windows.
- Do not simply lower dreamMs: that turns catch-up pressure into permanent steady-state churn.
- Do not add required deployment config: backlog catch-up is normal recovery behavior and should be enabled by default.
- Do not query Chroma every poll just to count backlog; use the prior saturated-batch signal for the cheap catch-up trigger.
Related
Parent epic: #13755
Prior context: discussion #12062 already documented the 10-session / one-hour throughput cap; discussion #13873 covers broader adaptive model residency and sweet-spot tuning.
Origin Session ID: 30ba1931-9d59-48bb-afdf-af8904070af9
Handoff Retrieval Hints: REM backlog catch-up scheduling saturated remSleepBatchLimit dreamMs; sessionsProcessed task state dream scheduler; issue 13755 REM heavy lease backlog drain.
Context
During local validation of the REM graph-extraction backlog, the orchestrator showed completion state behaving correctly but backlog drain behaving too slowly. At 2026-06-24T21:14:39Z, the latest 20 open issue sweep found no equivalent open ticket; the recent A2A claim sweep found no overlapping claim; local resources/content search found prior discussion analysis of the 10-per-hour cap but no actionable open child ticket.
The key observation: the 21:01 session summarization run completed and released its heavy-maintenance state at 21:08:11, so this is not an indefinitely stale running flag. The remaining defect is that REM goes back to the normal hourly cadence even when the previous REM pass saturated the configured work batch.
Release classification: boardless follow-up under #13755; not release-board scoped unless an operator explicitly retargets it.
The Problem
REM consolidation currently treats a large undigested-session backlog like normal steady-state maintenance. Default cadence is hourly, while the REM batch limit is 10 sessions per cycle. A current local REM run-state file showed sessionsProcessed: 10, exactly saturating the configured batch. With a backlog around 286 missing REM extractions, that implies roughly 29 successful hourly cycles before deferrals or competing heavy-maintenance lanes stretch it further.
This is too slow for a catch-up backlog. The system already knows enough to infer that more work probably remains: the completed REM cycle processed the maximum configured batch. That signal is recorded in REM run-state and health payloads, but it is not fed back into the scheduler due calculation.
The Architectural Reality
The result is a missing feedback edge: REM reports work saturation, but the scheduler cannot see it cheaply on the next poll.
The Fix
Add backlog-aware REM catch-up scheduling without lowering the normal hourly cadence:
Contract Ledger Matrix
Decision Record impact
Aligned with ADR 0019: config values must stay in AiConfig / existing Memory Core config leaves; no hidden defaults, no env re-derivation, and no config pass-through across modules beyond local leaf reads.
Acceptance Criteria
Out of Scope
Avoided Traps
Related
Parent epic: #13755
Prior context: discussion #12062 already documented the 10-session / one-hour throughput cap; discussion #13873 covers broader adaptive model residency and sweet-spot tuning.
Origin Session ID: 30ba1931-9d59-48bb-afdf-af8904070af9
Handoff Retrieval Hints: REM backlog catch-up scheduling saturated remSleepBatchLimit dreamMs; sessionsProcessed task state dream scheduler; issue 13755 REM heavy lease backlog drain.