LearnNewsExamplesServices
Frontmatter
id13624
title[Epic] Investigation: orchestrator heavy-maintenance runs 14h but the backlog doesn''t drain (session-summary 283→284) — net-progress, not fairness
stateClosed
labels
epicaiarchitectureperformancemodel-experience
assignees[]
createdAtJun 20, 2026, 3:29 PM
updatedAt3:21 PM
githubUrlhttps://github.com/neomjs/neo/issues/13624
authorneo-opus-grace
commentsCount12
parentIssuenull
subIssues
13637 Session-summary drift churn: same active sessions re-summarized every sweep
13638 miniSummary backfill never converges: per-turn inflow ≈ outflow + lease-starved
13639 Guard graph-node inserts outside UNIT_TEST_MODE (inserts have no guard twin)
13647 Re-scope pending-session-summary marker to the drainable set
13675 Hard-delete the test-pollution orphan sessions from the graph
13722 Make repairUnprojectedSessions dry-run truly read-only
13780 Heavy-lease cooperative-yield primitive: shouldYieldHeavyMaintenanceLease + maxActiveHoldMs knob
13807 Surface over-band/undigested sessions as a visible consolidation-gap in sandman_handoff.md
13818 REM consolidation-liveness watchdog — alarm on a stalled dream cycle
13834 Idle-window starvation: per-message concept-extraction fires ungated gemma4 calls in parallel with exclusive-heavy tasks
13835 Tri-state REM digest-state — stop re-serving un-digestible sessions
13839 Active alarm dispatcher for the REM consolidation-liveness watchdog
13840 MX-concept-harvester: scheduled message-concept drainer (frequency pre-filter → LLM Teaching-Test → dedup-by-name)
13850 REM session loop: per-session fault isolation — one throwing session aborts the whole batch (head-of-line blocking)
13851 fix(ai): orchestrator must verify a loaded model''s context window + free context, not just residency (gemma-4-31b resident at 4096 ≠ configured 131072 → all tri-vector extraction chokes → golden path starves)
13852 fix(ai): orchestrator has no Ollama serving lifecycle — no `ollama serve` ensure + no `num_ctx` preload (native-Ollama hits #13851''s context-starvation with zero handling)
13875 Extract configured task composition from Orchestrator
subIssuesCompleted17
subIssuesTotal17
contentTrust
projected
quarantined0
signals[]
blockedBy[]
blocking[]
closedAtJun 25, 2026, 2:02 AM

[Epic] Investigation: orchestrator heavy-maintenance runs 14h but the backlog doesn't drain (session-summary 283→284) — net-progress, not fairness

Closed v13.1.0/archive-v13-1-0-chunk-4 epicaiarchitectureperformancemodel-experience
neo-opus-grace
neo-opus-grace commented on Jun 20, 2026, 3:29 PM

The irony (why this is filed)

This nightshift the swarm spent 6h+ on agents who were busy but not shipping — the no-hold-state / watch-the-shipper reckoning (→ #13618 / #13623: the substance-over-performance teeth-test + the L-collab ratio-as-observability). ADR 0022 (#13604) came out of that arc. Running ADR-0022, the orchestrator is now the same failure one level down the stack: in 14h it switched heavy-maintenance tasks faithfully (ADR-0022 fairness works — no starvation) and logged real work (mini-summaries updated, 6 sessions summarized), but the open session-summary backlog went 283 → 284. Activity high, net-progress ≈ 0 — the watch-the-shipper trap, in code. ADR-0022 solved fairness (task-switching); the backlog needed net-progress accounting.

Evidence (operator, 2026-06-20)

  • 14h run; session-summary pending 283 → 284 (net negative).
  • pending-memory-minisummary ~3967 → ~3672 (slow; sometimes flat — inflow ≈ outflow during high add_memory activity).
  • Logs: session summarization, when it finally runs, finds candidates=7 (processed 6) while pending-session-summary=284 — the sweep reaches a tiny fraction of the reported pending.

Investigation angles (split into sub-tickets on pickup)

  1. Metric-truth / scope-mismatchpending-session-summary (284, all-time) vs the drift-detection sweep (30-day window → finds 7); the metric counts what the sweep structurally can't reach → a permanent stuck count. Prior-art (Grace memory): out-of-window >30d sessions the periodic sweep never touches + legacy sessions missing properties.sessionId = false-pending. Decide: align the metric to the reachable set · one-shot includeAll:true drain · or exclude-unreachable-from-metric.
  2. Garbage-session cleanup — unit-test-pollution sessions inflate the backlog + waste gemma4. Bare npx playwright test writes to the live graph (graphProd, not :memory:). Identify + hard-delete the garbage; guard the pollution path (UNIT_TEST_MODE).
  3. MiniSummary throughput vs inflow — backfill does ~90–105/run then completes early (why — lease time-budget? candidate cap?); every add_memory mints a new pending mini-summary → inflow ≈ outflow → never converges. Quantify throughput vs creation-rate.
  4. Parseability / completeness analysis daemon (operator) — a new daemon (or two) that checks whether sessions/memories can even be parsed / carry the needed mini-summaries → distinguish genuinely-pending from stuck / garbage / unparseable, so the backlog metric becomes truthful.
  5. gemma4 wasted work — gemma4 doing not-needed work (re-processing, garbage, out-of-window) → wasted local inference. Quantify + cut.

Framing AC

The fix-target is net backlog progress (the warrant), not task-switching activity (the costume). Apply the same substance-over-performance lens #13623 builds for agents to the orchestrator: fairness was necessary but insufficient.

Parent: #12065 (Orchestrator-as-SSOT for the REM Pipeline). Refs: ADR 0022 (#13604), #13592 (bounded summary lease-hold). Filed by @neo-opus-grace per @tobiu (2026-06-20) — operator-directed investigation; peers pick up the angles.

tobiu referenced in commit 64f13e5 - "feat(ai): hung-heavy-lease watchdog decision (pure) — sub of #13624 (#13760) (#13761) on Jun 21, 2026, 2:23 PM
tobiu referenced in commit deb90d3 - "feat(ai): hung-lease watchdog integration — orchestrator lease-monitor (sub of #13624) (#13768) on Jun 21, 2026, 4:00 PM
tobiu referenced in commit 17e9388 - "feat(ai): surface undigested sessions as a visible Consolidation Gaps section (#13807) (#13809) on Jun 21, 2026, 11:46 PM
tobiu referenced in commit 3bcc166 - "feat(ai): wire REM consolidation-liveness watchdog into the orchestrator (#13818) (#13838) on Jun 22, 2026, 5:08 AM