LearnNewsExamplesServices
Frontmatter
id12828
titleBackfill spin-loop on undrainable miniSummaries starves the heavy lane
stateClosed
labels
bugaiarchitectureperformance
assigneesneo-gpt
createdAtJun 9, 2026, 9:15 PM
updatedAtJun 9, 2026, 10:05 PM
githubUrlhttps://github.com/neomjs/neo/issues/12828
authorneo-opus-grace
commentsCount1
parentIssuenull
subIssues[]
subIssuesCompleted0
subIssuesTotal0
blockedBy[]
blocking[]
closedAtJun 9, 2026, 10:05 PM

Backfill spin-loop on undrainable miniSummaries starves the heavy lane

Closed v13.0.0/archive-v13-0-0-chunk-17 bugaiarchitectureperformance
neo-opus-grace
neo-opus-grace commented on Jun 9, 2026, 9:15 PM

Context

Live orchestrator logs (2026-06-09, post-restart, PID 2700) show the memory miniSummary backfill spin-looping: from 19:03:04 onward, Starting memory miniSummary backfill (pending-memory-minisummary:3333)completed successfully in under a second, repeated every ~3s, count frozen at 3333 — immediately after a normal run drained 3423 → 3333 ending 19:03:02. Simultaneously the session-summary backlog (387) stopped draining. Operator: "now turn summaries got stale too. no drain."

The Problem (two stalls, one root)

The backfill drained everything processable (5410 → 3333 across the session) then hit a wall at 3333. Sub-second "completed" with zero drain = zero model inference = the run selects nothing it can summarize. The residual 3333 are undrainable by the current backfill.

This is one root, not two bugs:

  • The count keeps reporting 3333 > 0, so the orchestrator re-triggers the backfill every ~3s — no no-progress backoff.
  • memory-summary-backfill is maintenanceClass: 'heavy' under exclusive-heavy backpressure, so the spinning backfill hogs the heavy-maintenance slot and starves the now-all-time session summarization (#12823) from ever draining the 299. So the 387 stalling and the 3333 stalling are the same bug.

The Architectural Reality

  • Count: getPendingMemorySummaryBackfillCount (ai/daemons/orchestrator/scheduling/memorySummaryBackfill.mjs) = COUNT(*) AGENT_MEMORY WHERE json_extract(...,'$.properties.miniSummary') IS NULL — all-time, counts undrainable nodes.
  • Drain: MemoryService.backfillMiniSummaries() (via ai/scripts/lifecycle/backfill-memory-summaries.mjs) drains only processable memories. Its MINI_SUMMARY_BACKFILL_MAX_RUN_MS JSDoc already half-anticipates a "loop without draining" mode — but for the slow-model case, not this undrainable-residual case.
  • Backpressure: heavy tasks are mutually exclusive (scheduling/picker.mjs filterExclusiveHeavyConflict; scheduling/registry.mjs backpressure: 'exclusive-heavy') — so the spinning backfill blocks session summarization from the slot.
  • Verified (logs + code): the spin-loop, the count, and the exclusive-heavy starvation mechanism. Hypothesis (needs confirmation): the 3333 residual are orphaned/content-less AGENT_MEMORY nodes (NULL miniSummary + missing/empty Chroma content) — the class of #12450. Not caused by #12823 (separate code path — confirmed by grep).

The Fix (three layers)

  1. No-progress backoff (urgent): a backfill run that drains 0 must not instantly reschedule — back off / mark the residual stuck so the heavy slot frees. This alone un-starves both pipelines.
  2. Honest count: exclude undrainable (content-less) nodes from getPendingMemorySummaryBackfillCount, or track attempted-but-failed, so the trigger stops firing forever on a residual that can never drain.
  3. Orphan cleanup: purge/tombstone the undrainable cohort (sibling to #12450; may live there).

Acceptance Criteria

  • Confirm the residual root: sample the NULL-miniSummary AGENT_MEMORY nodes and verify their Chroma content is missing/empty (or identify the actual undrainable reason).
  • A backfill run that drains 0 items does not immediately reschedule; the heavy-maintenance slot frees (no sub-second spin-loop in the logs).
  • Session summarization can acquire the heavy slot and drain while the backfill is backed off (the 387 + 299 drain).
  • The pending-memory-minisummary count stops counting permanently-undrainable nodes (or they are cleaned up), so the logged backlog reflects real drainable work.

Out of Scope

  • The orphaned/corrupt-vector data cleanup itself, if owned by #12450 (reference, don't duplicate).
  • The all-time session-summary change (#12823 — merged; this only un-starves it).

Avoided Traps

  • Just raising the watchdog / MAX_RUN_MS — doesn't fix a 0-drain spin; the run already completes fast. The issue is no progress + no backoff.
  • Deleting the count — loses a real backlog signal; make it honest, not blind.
  • Blaming #12823 — verified separate path; the all-time session summarization is the victim (starved), not the cause.

Decision Record impact

none — applies existing observability + scheduler-resilience patterns (no-progress backoff, honest metric).

Related

  • #12450 (corrupt sessions vectors — sibling for the orphan-cleanup layer)
  • #12823 (all-time session summarization — the starved victim, merged)
  • #12812 (intra-run backfill progress logs — would surface this faster)
  • #12740 (cost-safety epic context)

Origin Session ID: efc94c7f-3004-42e2-89cd-ed4102d9d2d7 Retrieval Hint: "miniSummary backfill spin-loop undrainable residual exclusive-heavy starvation no-progress backoff"

tobiu referenced in commit 77cb413 - "fix(orchestrator): back off stuck miniSummary windows (#12828) (#12829)" on Jun 9, 2026, 10:05 PM
tobiu closed this issue on Jun 9, 2026, 10:05 PM