LearnNewsExamplesServices
Frontmatter
id13971
titleREM backlog drains only at hourly steady-state cadence
stateClosed
labels
bugaiarchitectureperformancemodel-experience
assigneesneo-gpt
createdAtJun 24, 2026, 11:16 PM
updatedAtJun 24, 2026, 11:48 PM
githubUrlhttps://github.com/neomjs/neo/issues/13971
authorneo-gpt
commentsCount0
parentIssue13755
subIssues[]
subIssuesCompleted0
subIssuesTotal0
contentTrust
projected
quarantined0
signals[]
blockedBy[]
blocking[]
closedAtJun 24, 2026, 11:48 PM

REM backlog drains only at hourly steady-state cadence

Closed v13.1.0/archive-v13-1-0-chunk-6 bugaiarchitectureperformancemodel-experience
neo-gpt
neo-gpt commented on Jun 24, 2026, 11:16 PM

Context

During local validation of the REM graph-extraction backlog, the orchestrator showed completion state behaving correctly but backlog drain behaving too slowly. At 2026-06-24T21:14:39Z, the latest 20 open issue sweep found no equivalent open ticket; the recent A2A claim sweep found no overlapping claim; local resources/content search found prior discussion analysis of the 10-per-hour cap but no actionable open child ticket.

The key observation: the 21:01 session summarization run completed and released its heavy-maintenance state at 21:08:11, so this is not an indefinitely stale running flag. The remaining defect is that REM goes back to the normal hourly cadence even when the previous REM pass saturated the configured work batch.

Release classification: boardless follow-up under #13755; not release-board scoped unless an operator explicitly retargets it.

The Problem

REM consolidation currently treats a large undigested-session backlog like normal steady-state maintenance. Default cadence is hourly, while the REM batch limit is 10 sessions per cycle. A current local REM run-state file showed sessionsProcessed: 10, exactly saturating the configured batch. With a backlog around 286 missing REM extractions, that implies roughly 29 successful hourly cycles before deferrals or competing heavy-maintenance lanes stretch it further.

This is too slow for a catch-up backlog. The system already knows enough to infer that more work probably remains: the completed REM cycle processed the maximum configured batch. That signal is recorded in REM run-state and health payloads, but it is not fed back into the scheduler due calculation.

The Architectural Reality

  • ai/config.mjs:503 defines AiConfig.orchestrator.intervals.dreamMs as the normal hourly REM cadence.
  • ai/mcp/server/memory-core/config.mjs:145 defines remSleepBatchLimit as 10 sessions per REM cycle.
  • ai/daemons/orchestrator/scheduling/dream.mjs:54 only gates the next REM run by elapsed time since the cadence anchor.
  • ai/daemons/orchestrator/scheduling/pipeline.mjs:477 builds a recordPayload with sessionsProcessed, but pipeline.mjs:487 calls taskStateService.markCompleted(taskName) without persisting that work signal into task state.
  • ai/daemons/orchestrator/services/TaskStateService.mjs:190 only records success timing and exit code on completion.

The result is a missing feedback edge: REM reports work saturation, but the scheduler cannot see it cheaply on the next poll.

The Fix

Add backlog-aware REM catch-up scheduling without lowering the normal hourly cadence:

  • Persist REM completion metadata on the dream task state, at minimum sessionsProcessed and whether the completed batch saturated remSleepBatchLimit.
  • Extend the pure dream scheduler so a successful, non-overflowed, saturated REM cycle becomes due again after a short catch-up cooldown.
  • Return to normal hourly cadence when the previous cycle was skipped, failed, non-saturated, or overflowed.
  • Keep the heavy-maintenance lease invariant: no parallel REM cycles and no competing heavy task overlap.
  • Keep defaults working out of the box; no deployment env should be required.

Contract Ledger Matrix

Target Surface Source of Authority Proposed Behavior Fallback Docs Evidence
Dream task state TaskStateService + orchestrator task state JSON Persist bounded REM outcome metadata needed by scheduler catch-up Existing timing-only state remains readable JSDoc on new state helper / fields Unit test asserts metadata persistence
Dream scheduler ai/daemons/orchestrator/scheduling/dream.mjs Schedule short-cooldown catch-up only after saturated successful non-overflow REM cycles Normal dreamMs cadence JSDoc on due-trigger policy Unit tests for saturated, non-saturated, skipped/failed/overflow paths
REM service outcome DreamService.executeRemCycle() outcome envelope Continue emitting sessionsProcessed; caller derives saturation against config-owned batch limit Existing outcome shape stays compatible Existing run-state docs/comments Unit or focused service test uses synthetic outcome

Decision Record impact

Aligned with ADR 0019: config values must stay in AiConfig / existing Memory Core config leaves; no hidden defaults, no env re-derivation, and no config pass-through across modules beyond local leaf reads.

Acceptance Criteria

  • A saturated successful REM cycle schedules a catch-up run before the normal dreamMs interval expires.
  • A non-saturated successful REM cycle returns to the normal dreamMs cadence.
  • Skipped, failed, and overflowed REM cycles do not enter catch-up churn.
  • Catch-up mode keeps a single heavy-maintenance lease and never runs REM in parallel.
  • Defaults remain deployable without new env vars.
  • Focused unit coverage proves scheduler/state behavior.

Out of Scope

  • Changing REM extraction semantics, chunking, tokenizer guards, or graph node schema.
  • Increasing local model parallelism or changing provider model selection.
  • Lowering the normal hourly REM cadence globally.
  • Solving adaptive downscaling / hardware sweet-spot tuning from discussion #13873.

Avoided Traps

  • Do not simply raise remSleepBatchLimit: that makes each heavy lease larger and can recreate long blocking windows.
  • Do not simply lower dreamMs: that turns catch-up pressure into permanent steady-state churn.
  • Do not add required deployment config: backlog catch-up is normal recovery behavior and should be enabled by default.
  • Do not query Chroma every poll just to count backlog; use the prior saturated-batch signal for the cheap catch-up trigger.

Related

Parent epic: #13755

Prior context: discussion #12062 already documented the 10-session / one-hour throughput cap; discussion #13873 covers broader adaptive model residency and sweet-spot tuning.

Origin Session ID: 30ba1931-9d59-48bb-afdf-af8904070af9

Handoff Retrieval Hints: REM backlog catch-up scheduling saturated remSleepBatchLimit dreamMs; sessionsProcessed task state dream scheduler; issue 13755 REM heavy lease backlog drain.

tobiu referenced in commit 6982eb4 - "fix(ai): add REM backlog catch-up scheduling (#13971) (#13972)" on Jun 24, 2026, 11:48 PM
tobiu closed this issue on Jun 24, 2026, 11:48 PM