miniSummary backfill re-fires forever — no-progress backoff is un-armable

Context

Operator-flagged 2026-06-12: "there should never be 1000s of repeating logs… the repetition is [the problem]." The memory-summary-backfill task re-fires continuously and holds the exclusive heavy-maintenance lane, starving both session summarization (395 pending) and REM digestion (undigested: 362, recentCycles: [] via get_rem_pipeline_state, 01:58Z). The logs are exhaust; the repetition is the fault.

The Problem — verified root cause

The no-progress backoff (ai/daemons/orchestrator/scheduling/memorySummaryBackfill.mjs) is structurally un-armable while memory writes flow.

• It arms only when a run is "stalled": buildNoProgressBackoffHook (line 131) → stalled = (afterCount === beforeCount) && samePendingWindow(afterJobs, pendingJobs).
• The pending window is WHERE miniSummary IS NULL ORDER BY timestamp DESC LIMIT 50 (line 33) — the newest 50 un-summarized rows.
• Every agent turn writes a new AGENT_MEMORY with miniSummary: NULL (the per-turn add_memory). It lands at the DESC top → it both bumps afterCount (afterCount !== beforeCount) AND shifts the newest-50 window (samePendingWindow → false).
• → stalled is false → the backoff is cleared (143-146) → getDueTask (202) sees no active backoff → it re-fires immediately.
• On a busy session, the backoff can never arm → tight re-fire loop → 1000s of repeating logs + lane starvation.

The design names window-change→run as intentional (83-84: "If new memories arrive above the stuck residual, the window changes and the scheduler lets the backfill run immediately"). The flaw is its collision with a second condition: the newest rows can't be summarized — MemoryService.backfillMiniSummaries returns missingContent for them (line 1034; Chroma metadata has no prompt/response). So the un-summarizable head never drains while new arrivals constantly reset the backoff. The escape hatch is the trap.

Hypothesis (UNVERIFIED — not yet sampled): the missingContent head is the async Chroma content-embed not having landed yet (add_memory writes the graph; content embeds later, MemoryService.mjs:297). Confirm before relying on it.

The Architectural Reality (verified surfaces)

• Scheduler/backoff: ai/daemons/orchestrator/scheduling/memorySummaryBackfill.mjs — samePendingWindow (75), isNoProgressBackoffActive (92), buildNoProgressBackoffHook (stalled @ 131), getDueTask (190), window query (28-35).
• Backfill body: ai/services/memory-core/MemoryService.mjs:974-1069 — newest-first select (987-995), missingContent (1034), summarize-null defer (1047), run-budget break (1017).
• Chroma is UP (MC healthcheck: memories 18,388 records, connected) — this is NOT a content-store outage; the bug is the backoff logic. (The KB's separate knowledgeBase collection outage is a different ticket.)

The Fix — three layers

1. Arm the backoff on real progress (updated === 0), not window-identity. Zero summaries written = no progress, regardless of new arrivals. Decouple "fresh row arrived" from "a summary was written"; fresh rows can have a cheap fast-lane that doesn't hold the heavy mutex.
2. Stop re-selecting un-summarizable heads. Skip rows with no Chroma content, or order oldest-first to drain the embeddable backlog instead of re-chewing the un-embeddable newest.
3. Yield the heavy-maintenance mutex on no-progress so a 0-updated task can't starve session summaries + REM digestion.

Acceptance Criteria

• A run writing 0 summaries arms the no-progress backoff even when new miniSummary: NULL rows arrived (regression test: write a memory between runs; assert backoff stays armed).
• The backfill no longer re-fires every scheduler tick while the head is un-summarizable; it backs off for NO_PROGRESS_BACKOFF_MS.
• Session summarization + REM digestion get the lane while the backfill is in no-progress backoff.
• (Investigate) confirm embed-lag vs permanently-contentless for the missingContent head.

Out of Scope

• The KB knowledgeBase collection outage (ask_knowledge_base "Failed to access collections") — separate ticket.
• Temporal-Pyramid / hierarchical-summarization strategy (#12679 / #12073).

Related

• Parent area: #12065 ([Epic] Orchestrator-as-SSOT for the REM (Sandman) Pipeline).

Release classification: boardless — Agent-OS infra bug; not v13-release-gating, but an active production degradation (memory pipeline starved now). Immediate mitigation: pause the memory-summary-backfill task to free the lane.

Origin Session ID: 0bc22bf2-9ded-4cdb-b351-182c3ab1d16f

Retrieval Hint: "miniSummary backfill no-progress backoff un-armable samePendingWindow newest-50 re-fire forever heavy-maintenance starve REM session summary"