Heavy-maintenance lease monopoly starves backup, backfill + REM

Context

Operator-reported, 2026-06-20: even after #13579 merged (which stopped the session-summary re-loop), the live post-restart orchestrator log shows session summarization again monopolizing the heavy-maintenance lease and deferring the entire rest of the pipeline:

[02:48:58] Starting session summarization (pending-session-summary:283)
[02:49:01] Deferring memory miniSummary backfill; heavy maintenance task session summarization is active (pending-memory-minisummary:4415)
[02:49:01] Deferring agent OS backup; ... session summarization is active
[02:49:01] Deferring REM sleep graph extraction; ... session summarization is active
[02:49:01] Deferring knowledge base sync / GraphLog compaction / primary dev sync; ... active
[03:01:01] session summarization: 1/8 done

~12 min/session and only 1/8 done after 12 minutes → the lease is held for the whole batch (~90+ min), during which nothing else heavy runs: 4,415 pending miniSummaries, 371 undigested REM summaries, and — most critically — the daily backup is deferred. Operator: "daily backup saving must be prio 0 … if that gets pushed away => DEEP trouble" (a missed daily backup = data-loss exposure for the live deployment).

The Problem

The exclusive-heavy mutex (one substrate-heavy job at a time) is correct — all these tasks hammer the single local gemma4, so serializing them is right. The bug is unfairness on top of the mutex:

1. Fixed registry-order priority. pickNextCandidate (scheduling/picker.mjs:111) ends in selectFirstCandidate — "registry order is the priority." In TASK_REGISTRY (scheduling/registry.mjs:36), summary is registry-first and backup is 4th. So whenever summary is due (every 600 s) and the lease is free, summary wins — backup / memory-summary-backfill / dream (REM) all lose.
2. Unbounded lease-hold. The summary child drains the entire findSessionsToSummarize list in one run (SessionService.mjs:1386, for (i=0; i<total; i+=batchSize)), holding the lease start-to-finish. So even a task that would win a fair pick never gets the chance — the lease is held for the whole batch.

Together: any summary backlog starves the whole heavy pipeline indefinitely, and the daily backup can miss its window entirely.

Distinct from #12943 (CLOSED — the backfill's own no-progress backoff arming) and complementing #12065 (the REM orchestrator-SSOT epic): #12943 made the backfill yield to summary; this ticket stops summary from monopolizing.

The Architectural Reality

• Orchestrator.poll() → runSchedulingPipeline (scheduling/pipeline.mjs) → collectDueCandidates (scheduling/collector.mjs, iterates TASK_REGISTRY) → pickNextCandidate (scheduling/picker.mjs) selects ONE winner → executeCandidate → MaintenanceBackpressureService.acquireLeaseAndExecute (the exclusive-heavy lease).
• TASK_REGISTRY order (registry.mjs:36): summary(1), memory-summary-backfill(2), kbSync(3), backup(4), graphlog-compaction, primary-dev-sync, tenant-repo-sync, dream(REM), golden-path, swarm-heartbeat.
• The picker is pure (picker.mjs:29) — filters (already-running, exclusive-heavy-conflict, unmet-dependencies) then selectFirstCandidate. The final selection is the only place priority is encoded → the clean seam for fairness.
• DEFAULT_HEAVY_MAINTENANCE_TASK_NAMES (MaintenanceBackpressureService.mjs:18) defines mutex membership; DEFAULT_COMPATIBLE_HEAVY_MAINTENANCE_TASK_PAIRS (only kbSync+memory-summary-backfill) the sole concurrent pair.

The Fix

Part 1 — Fair picker (scheduling/picker.mjs): replace selectFirstCandidate with a priority + fairness selector:

• Backup = priority 0: if backup survives the filters, it wins unconditionally (operator requirement — a daily backup must never be deferred). Least-recently-run alone would starve the backup, since the never-run backfill always has an older lastRunAt, so backup needs an explicit priority-0 special-case, not just LRR.
• Least-recently-run for the rest: among the remaining heavy survivors, pick the oldest lastRunAt — the starved backfill (never run) and REM beat summary (runs every 600 s). Non-heavy tasks keep registry order.
• Plumb each candidate's lastRunAt (from TaskStateService) into the picker via policyContext — keeps the picker pure.

Part 2 — Bounded summary lease-hold (SessionService.mjs:1386): cap sessions-per-sweep to a small config-driven maxSessionsPerSweep. The child processes ≤ N sessions, exits + releases the lease; the next sweep continues the backlog. Required for Part 1 — without it, summary holds the lease for the whole batch and the fair picker never gets to choose.

Backup approach: bounded-hold + priority-0 (backup waits at most one short summary chunk — minutes — then wins), NOT a hard pre-empt that kills a running summary child mid-flight (safer; minutes of delay on a daily backup is harmless).

Acceptance Criteria

• picker.mjs: backup, when a surviving candidate, is selected ahead of any other heavy candidate (priority-0). Unit-tested.
• picker.mjs: among non-backup heavy survivors, the oldest-lastRunAt candidate is selected (least-recently-run), NOT registry order — e.g. a starved memory-summary-backfill beats a recently-run summary. Unit-tested.
• policyContext carries per-candidate lastRunAt; the picker stays pure (no state writes). Unit-tested.
• SessionService drift summarization caps sessions-per-run to a bounded config-driven N; the child exits + releases the lease after N; the remainder persists for the next sweep. Unit-tested.
• Non-heavy / continuous / dependency behavior unchanged (the new selector only re-orders heavy candidates); existing picker/pipeline tests stay green.
• Post-merge (flagged): on a live orchestrator with a summary backlog, the deferral log no longer shows backup / memory-summary-backfill / dream deferred for more than one summary chunk; pending-memory-minisummary drops and recentCycles populates.

Out of Scope

• The mutex itself (serializing the single-gemma4 heavy tasks is correct — untouched).
• The backfill's internal no-progress backoff (#12943, already closed).
• The newest-first DESC backfill window (separate follow-up: it can lag the oldest NULLs even once it runs).
• Per-session summarization speed (the ~12 min/session is a separate inference-throughput concern).

Avoided Traps

• Hard pre-empt of a running summary for the backup — rejected: killing a heavy child mid-flight risks partial state; bounded-hold + priority-0 lands the backup within minutes without that risk.
• Pure least-recently-run for everything (no backup special-case) — rejected: the never-run backfill always out-ages the backup, so LRR alone would starve the very task flagged as priority-0.
• Registry reorder alone (move backup to front) — insufficient: fixes backup but leaves backfill/REM starved behind summary, and doesn't address the unbounded hold.

Related

• #12943 (CLOSED) — miniSummary backfill no-progress backoff (the backfill yield side; this ticket is the summary monopoly side).
• #12065 (OPEN epic) — Orchestrator-as-SSOT for the REM pipeline (sibling orchestrator concern).
• #13579 (MERGED) — stopped the session-summary re-loop (shrank the backlog but didn't touch priority/hold; this ticket is the remaining root).

Decision Record impact

none — a scheduling-policy fix within the existing orchestrator architecture; the mutex + registry model is preserved.

Release classification: post-v13 agent-OS stability hardening — high priority (daily-backup starvation = data-loss exposure for the live deployment). Likely board #13 (Agent Harness); filed boardless for operator/lead placement.

Origin Session ID: fca648cb-7dbf-418a-b2d2-498df8e0213c

Retrieval Hint: "heavy-maintenance lease monopoly fair picker backup priority-0 least-recently-run" / picker.mjs selectFirstCandidate / SessionService:1386 bounded summary hold