Problem
The memory-summary-backfill maintenance task (orchestrator → supervised lifecycle child running MemoryService.backfillMemoryMiniSummaries) can hang indefinitely on un-timeout-guarded downstream calls.
Observed live 2026-06-07: the backfill went "active" at 22:08:55 for a 50-item batch and was still active 4h+ later (a 50-item batch should take minutes), with zero fail-soft warnings in the MC log — i.e. hung-awaiting, not erroring. While its "active" flag is set, the orchestrator defers ALL other maintenance (session-summarization + GraphLog-compaction observed deferred at 22:14 and 23:35, then silence). Chroma was healthy throughout (heartbeat 200 in 0.5ms), so the model call is the prime hang site.
Root cause — ai/services/memory-core/MemoryService.mjs backfill loop
summarize() (≈L806) → this.buildMiniSummary({prompt, response}) (a model call) has no timeout. The per-memory try/catch (≈L805-824) catches errors, but a hung await never throws → it sits forever (no fail-soft warning).
- The Chroma fetch (≈L792-793) —
StorageRouter.getMemoryCollection() + collection.get(...) — is outside the per-memory try/catch, so a stall/error there isn't even fail-soft; it hangs/propagates the whole task.
- No supervisor watchdog:
ai/daemons/orchestrator/services/ProcessSupervisorService.mjs has PID-file recovery but no max-runtime on a live spawned child, so one hung child blocks the entire maintenance loop until a manual restart.
Acceptance Criteria
Impact
No data loss (the MC still serves MCP calls), but maintenance stalls indefinitely until a manual orchestrator restart. Substrate-resilience gap: unbounded-await + unbounded-defer = one hung child wedges the whole maintenance loop.
Refs #12065 (Orchestrator-as-SSOT for the REM/Sandman pipeline).
Authored by neo-opus-grace (diagnosed live from the orchestrator-log stall @tobiu flagged).
Problem
The
memory-summary-backfillmaintenance task (orchestrator → supervised lifecycle child runningMemoryService.backfillMemoryMiniSummaries) can hang indefinitely on un-timeout-guarded downstream calls.Observed live 2026-06-07: the backfill went "active" at 22:08:55 for a 50-item batch and was still active 4h+ later (a 50-item batch should take minutes), with zero fail-soft warnings in the MC log — i.e. hung-awaiting, not erroring. While its "active" flag is set, the orchestrator defers ALL other maintenance (session-summarization + GraphLog-compaction observed deferred at 22:14 and 23:35, then silence). Chroma was healthy throughout (heartbeat 200 in 0.5ms), so the model call is the prime hang site.
Root cause —
ai/services/memory-core/MemoryService.mjsbackfill loopsummarize()(≈L806) →this.buildMiniSummary({prompt, response})(a model call) has no timeout. The per-memorytry/catch(≈L805-824) catches errors, but a hungawaitnever throws → it sits forever (no fail-soft warning).StorageRouter.getMemoryCollection()+collection.get(...)— is outside the per-memorytry/catch, so a stall/error there isn't even fail-soft; it hangs/propagates the whole task.ai/daemons/orchestrator/services/ProcessSupervisorService.mjshas PID-file recovery but no max-runtime on a live spawned child, so one hung child blocks the entire maintenance loop until a manual restart.Acceptance Criteria
summarize()/buildMiniSummarycall in the backfill loop is wrapped with a timeout → a hung model call fails-soft via the existing L822 path and the loop continues.getMemoryCollection/collection.get) is wrapped in try/catch + timeout → a stall defers the batch gracefully instead of hanging.ProcessSupervisorServiceenforces a per-task max-runtime watchdog: a child exceeding the bound is killed + its active flag cleared, so one hung child can never block all maintenance indefinitely.Impact
No data loss (the MC still serves MCP calls), but maintenance stalls indefinitely until a manual orchestrator restart. Substrate-resilience gap: unbounded-await + unbounded-defer = one hung child wedges the whole maintenance loop.
Refs #12065 (Orchestrator-as-SSOT for the REM/Sandman pipeline). Authored by neo-opus-grace (diagnosed live from the orchestrator-log stall @tobiu flagged).