Context
Live orchestrator logs (2026-06-09, post-restart, PID 2700) show the memory miniSummary backfill spin-looping: from 19:03:04 onward, Starting memory miniSummary backfill (pending-memory-minisummary:3333) → completed successfully in under a second, repeated every ~3s, count frozen at 3333 — immediately after a normal run drained 3423 → 3333 ending 19:03:02. Simultaneously the session-summary backlog (387) stopped draining. Operator: "now turn summaries got stale too. no drain."
The Problem (two stalls, one root)
The backfill drained everything processable (5410 → 3333 across the session) then hit a wall at 3333. Sub-second "completed" with zero drain = zero model inference = the run selects nothing it can summarize. The residual 3333 are undrainable by the current backfill.
This is one root, not two bugs:
- The count keeps reporting
3333 > 0, so the orchestrator re-triggers the backfill every ~3s — no no-progress backoff.
memory-summary-backfill is maintenanceClass: 'heavy' under exclusive-heavy backpressure, so the spinning backfill hogs the heavy-maintenance slot and starves the now-all-time session summarization (#12823) from ever draining the 299. So the 387 stalling and the 3333 stalling are the same bug.
The Architectural Reality
- Count:
getPendingMemorySummaryBackfillCount (ai/daemons/orchestrator/scheduling/memorySummaryBackfill.mjs) = COUNT(*) AGENT_MEMORY WHERE json_extract(...,'$.properties.miniSummary') IS NULL — all-time, counts undrainable nodes.
- Drain:
MemoryService.backfillMiniSummaries() (via ai/scripts/lifecycle/backfill-memory-summaries.mjs) drains only processable memories. Its MINI_SUMMARY_BACKFILL_MAX_RUN_MS JSDoc already half-anticipates a "loop without draining" mode — but for the slow-model case, not this undrainable-residual case.
- Backpressure: heavy tasks are mutually exclusive (
scheduling/picker.mjs filterExclusiveHeavyConflict; scheduling/registry.mjs backpressure: 'exclusive-heavy') — so the spinning backfill blocks session summarization from the slot.
- Verified (logs + code): the spin-loop, the count, and the exclusive-heavy starvation mechanism. Hypothesis (needs confirmation): the 3333 residual are orphaned/content-less
AGENT_MEMORY nodes (NULL miniSummary + missing/empty Chroma content) — the class of #12450. Not caused by #12823 (separate code path — confirmed by grep).
The Fix (three layers)
- No-progress backoff (urgent): a backfill run that drains 0 must not instantly reschedule — back off / mark the residual stuck so the heavy slot frees. This alone un-starves both pipelines.
- Honest count: exclude undrainable (content-less) nodes from
getPendingMemorySummaryBackfillCount, or track attempted-but-failed, so the trigger stops firing forever on a residual that can never drain.
- Orphan cleanup: purge/tombstone the undrainable cohort (sibling to #12450; may live there).
Acceptance Criteria
Out of Scope
- The orphaned/corrupt-vector data cleanup itself, if owned by #12450 (reference, don't duplicate).
- The all-time session-summary change (#12823 — merged; this only un-starves it).
Avoided Traps
- Just raising the watchdog /
MAX_RUN_MS — doesn't fix a 0-drain spin; the run already completes fast. The issue is no progress + no backoff.
- Deleting the count — loses a real backlog signal; make it honest, not blind.
- Blaming #12823 — verified separate path; the all-time session summarization is the victim (starved), not the cause.
Decision Record impact
none — applies existing observability + scheduler-resilience patterns (no-progress backoff, honest metric).
Related
- #12450 (corrupt sessions vectors — sibling for the orphan-cleanup layer)
- #12823 (all-time session summarization — the starved victim, merged)
- #12812 (intra-run backfill progress logs — would surface this faster)
- #12740 (cost-safety epic context)
Origin Session ID: efc94c7f-3004-42e2-89cd-ed4102d9d2d7
Retrieval Hint: "miniSummary backfill spin-loop undrainable residual exclusive-heavy starvation no-progress backoff"
Context
Live orchestrator logs (2026-06-09, post-restart, PID 2700) show the memory miniSummary backfill spin-looping: from
19:03:04onward,Starting memory miniSummary backfill (pending-memory-minisummary:3333)→completed successfullyin under a second, repeated every ~3s, count frozen at 3333 — immediately after a normal run drained3423 → 3333ending19:03:02. Simultaneously the session-summary backlog (387) stopped draining. Operator: "now turn summaries got stale too. no drain."The Problem (two stalls, one root)
The backfill drained everything processable (
5410 → 3333across the session) then hit a wall at 3333. Sub-second "completed" with zero drain = zero model inference = the run selects nothing it can summarize. The residual 3333 are undrainable by the current backfill.This is one root, not two bugs:
3333 > 0, so the orchestrator re-triggers the backfill every ~3s — no no-progress backoff.memory-summary-backfillismaintenanceClass: 'heavy'underexclusive-heavybackpressure, so the spinning backfill hogs the heavy-maintenance slot and starves the now-all-time session summarization (#12823) from ever draining the 299. So the387stalling and the3333stalling are the same bug.The Architectural Reality
getPendingMemorySummaryBackfillCount(ai/daemons/orchestrator/scheduling/memorySummaryBackfill.mjs) =COUNT(*) AGENT_MEMORY WHERE json_extract(...,'$.properties.miniSummary') IS NULL— all-time, counts undrainable nodes.MemoryService.backfillMiniSummaries()(viaai/scripts/lifecycle/backfill-memory-summaries.mjs) drains only processable memories. ItsMINI_SUMMARY_BACKFILL_MAX_RUN_MSJSDoc already half-anticipates a "loop without draining" mode — but for the slow-model case, not this undrainable-residual case.scheduling/picker.mjsfilterExclusiveHeavyConflict;scheduling/registry.mjsbackpressure: 'exclusive-heavy') — so the spinning backfill blocks session summarization from the slot.AGENT_MEMORYnodes (NULLminiSummary + missing/empty Chroma content) — the class of #12450. Not caused by #12823 (separate code path — confirmed by grep).The Fix (three layers)
getPendingMemorySummaryBackfillCount, or track attempted-but-failed, so the trigger stops firing forever on a residual that can never drain.Acceptance Criteria
NULL-miniSummaryAGENT_MEMORYnodes and verify their Chroma content is missing/empty (or identify the actual undrainable reason).387+ 299 drain).pending-memory-minisummarycount stops counting permanently-undrainable nodes (or they are cleaned up), so the logged backlog reflects real drainable work.Out of Scope
Avoided Traps
MAX_RUN_MS— doesn't fix a 0-drain spin; the run already completes fast. The issue is no progress + no backoff.Decision Record impact
none— applies existing observability + scheduler-resilience patterns (no-progress backoff, honest metric).Related
Origin Session ID: efc94c7f-3004-42e2-89cd-ed4102d9d2d7 Retrieval Hint: "miniSummary backfill spin-loop undrainable residual exclusive-heavy starvation no-progress backoff"