LearnNewsExamplesServices
Frontmatter
id12711
titleMemory miniSummary backfill can hang indefinitely (no timeouts), wedging all orchestrator maintenance
stateClosed
labels
bugaiarchitecture
assigneesneo-opus-grace
createdAtJun 8, 2026, 2:17 AM
updatedAtJun 8, 2026, 3:14 AM
githubUrlhttps://github.com/neomjs/neo/issues/12711
authorneo-opus-grace
commentsCount0
parentIssuenull
subIssues[]
subIssuesCompleted0
subIssuesTotal0
blockedBy[]
blocking[]
closedAtJun 8, 2026, 3:14 AM

Memory miniSummary backfill can hang indefinitely (no timeouts), wedging all orchestrator maintenance

Closed v13.0.0/archive-v13-0-0-chunk-17 bugaiarchitecture
neo-opus-grace
neo-opus-grace commented on Jun 8, 2026, 2:17 AM

Problem

The memory-summary-backfill maintenance task (orchestrator → supervised lifecycle child running MemoryService.backfillMemoryMiniSummaries) can hang indefinitely on un-timeout-guarded downstream calls.

Observed live 2026-06-07: the backfill went "active" at 22:08:55 for a 50-item batch and was still active 4h+ later (a 50-item batch should take minutes), with zero fail-soft warnings in the MC log — i.e. hung-awaiting, not erroring. While its "active" flag is set, the orchestrator defers ALL other maintenance (session-summarization + GraphLog-compaction observed deferred at 22:14 and 23:35, then silence). Chroma was healthy throughout (heartbeat 200 in 0.5ms), so the model call is the prime hang site.

Root cause — ai/services/memory-core/MemoryService.mjs backfill loop

  1. summarize() (≈L806)this.buildMiniSummary({prompt, response}) (a model call) has no timeout. The per-memory try/catch (≈L805-824) catches errors, but a hung await never throws → it sits forever (no fail-soft warning).
  2. The Chroma fetch (≈L792-793)StorageRouter.getMemoryCollection() + collection.get(...) — is outside the per-memory try/catch, so a stall/error there isn't even fail-soft; it hangs/propagates the whole task.
  3. No supervisor watchdog: ai/daemons/orchestrator/services/ProcessSupervisorService.mjs has PID-file recovery but no max-runtime on a live spawned child, so one hung child blocks the entire maintenance loop until a manual restart.

Acceptance Criteria

  • The summarize()/buildMiniSummary call in the backfill loop is wrapped with a timeout → a hung model call fails-soft via the existing L822 path and the loop continues.
  • The Chroma fetch (getMemoryCollection/collection.get) is wrapped in try/catch + timeout → a stall defers the batch gracefully instead of hanging.
  • ProcessSupervisorService enforces a per-task max-runtime watchdog: a child exceeding the bound is killed + its active flag cleared, so one hung child can never block all maintenance indefinitely.
  • A test covers the timeout→fail-soft path with no real model/Chroma dependency.

Impact

No data loss (the MC still serves MCP calls), but maintenance stalls indefinitely until a manual orchestrator restart. Substrate-resilience gap: unbounded-await + unbounded-defer = one hung child wedges the whole maintenance loop.

Refs #12065 (Orchestrator-as-SSOT for the REM/Sandman pipeline). Authored by neo-opus-grace (diagnosed live from the orchestrator-log stall @tobiu flagged).

tobiu closed this issue on Jun 8, 2026, 3:14 AM
tobiu referenced in commit 5139e35 - "fix(ai): timeout-guard miniSummary backfill + orchestrator task watchdog (#12711) (#12712) on Jun 8, 2026, 3:14 AM