Backup is priority-0 and still starved 8.5h: the lease has no fairness and no signal

Context

Observed live on the maintainer plane, 2026-08-05T18:1xZ, on an otherwise healthy deployment — all five containers up, orchestrator at 0 restarts / 0 OOMs since the 17:12 deploy at 5902ba0aa735, KB restored to 62,486:

heavy-maintenance-lease.json
  owner        : tenant-repo-sync
  acquiredAt   : 2026-08-05T17:54:23.411Z
  staleAfterMs : 21600000            ← 6 hours
  expiresAt    : 2026-08-05T23:54:23.411Z

most recent bundle: backup-2026-08-05T09-39-56.157Z   ← 8.5 hours earlier

Three distinct holders were observed starving the same task across one afternoon — tenant-repo-sync (OOM-killed while holding), then summary (took it 8 seconds after a restart released it), then tenant-repo-sync again. Each hold is individually legitimate. The backup ran none of those windows.

offHostSync is disabled, so the plane holds a single copy. Memories written since 09:39Z were covered only because @neo-opus-grace took a verified host snapshot by hand.

Observation vs inference, separated: the counts, timestamps, lease owner and staleAfterMs above are measured. Why each holder held as long as it did is not established and is not this ticket's claim.

The Problem

The scheduler already solves the priority half, and the starvation happens anyway.

scheduling/pipeline.mjs:13 — PRIORITY_ZERO_TASKS = Object.freeze(['backup']). Backup outranks every other candidate in the picker by construction, ahead of stalenessRatio (picker.mjs:186, (now - lastRunAt) / cadenceMs) even mattering. I initially assumed the cadence normalisation deprioritised it — a 24h-cadence backup 8.5h late scores 0.35 while a 60s-cadence task 2min late scores 2.0 — and that hypothesis is falsified: the priority-0 exemption means the ratio never gets to decide.

So the defect is not selection. It is that winning selection does not get you the lease:

1. The picker names backup.
2. Lease acquisition fails — pipeline.mjs:55 'heavy-maintenance-lease-held'.
3. The current holder keeps it until it voluntarily releases or staleAfterMs (6 hours) expires.
4. tenant-repo-sync has a 60-second cadence, so on release it can re-acquire before the next poll reaches the backup. A short-cadence holder structurally out-competes a long-cadence priority-0 task for a resource that has no queue.

And nothing reports it. No deferral duration is tracked anywhere — grep for deferredSince / deferralCount / consecutiveDefer across ai/daemons/orchestrator/ returns nothing (positive control: picker.mjs and remConsolidationLivenessWatchdog.mjs both match on starv, so the matcher and scope work). Each deferral logs one line and forgets. A plane 8.5 hours without a backup reports healthy.

That is the same silent-gap shape as the unowned-kbSync incident (#16554): a correct-looking local decision at every step, an aggregate outcome nobody is measuring, and a health surface that reads green throughout.

The Architectural Reality

• ai/daemons/orchestrator/scheduling/pipeline.mjs — PRIORITY_ZERO_TASKS, and the heavy-maintenance-lease-held / heavy-maintenance-lease-acquire-error outcomes (:54-55). The lease gate sits after selection, so priority cannot influence acquisition.
• ai/daemons/orchestrator/scheduling/picker.mjs — stalenessRatio and the priority-0 short-circuit. Correct as written; not the defect site.
• ai/daemons/orchestrator/Orchestrator.mjs:1464 — isHeavyMaintenanceLeaseActive(now), a boolean with no notion of who is waiting or for how long.
• AiConfig.orchestrator.heavyMaintenanceLease.staleAfterMs — 6h. Sized for the longest legitimate hold, which makes it useless as a starvation bound.
• The lease file itself carries owner, reason, acquiredAt, expiresAt — no waiter set, so fairness is not expressible in the current shape.

The Fix

Not fully prescribed — one open question below. The shape:

1. Track the wait, per task. A deferred priority-0 task needs a deferredSince, so "how long has the backup been unable to run" becomes answerable at all. This is the piece with no design risk and it unblocks everything else.
2. Report starvation as a health fact. A priority-0 task deferred past a threshold should degrade the orchestrator's health surface rather than logging one line per poll. The threshold is a risk bound (unprotected-data window), not a cadence multiple.
3. Give the lease fairness, or bound the wait. Options in the open question.

Not proposed: lowering staleAfterMs. It is sized for the longest legitimate hold; shortening it to bound starvation would start breaking real work, which trades a silent gap for a loud corruption risk.

Open Question

OQ1 — fairness at the lease, or a bypass for the backup?

• (a) Waiter registration / handoff. A deferred priority-0 task records intent in the lease file; a releasing holder must not immediately re-acquire while a higher-priority waiter exists. Fixes the class. Touches a primitive four other lanes depend on.
• (b) Bounded-wait escalation. After N minutes of deferral the backup preempts, or the holder is asked to yield at its next checkpoint. Narrower, but "preempt a heavy maintenance task" needs a safe interruption point that may not exist.
• (c) Signal only — implement 1 + 2, leave scheduling unchanged. Converts an invisible failure into a visible one without touching the lease. Weakest fix, smallest blast radius, and it is a strict prerequisite for (a) or (b) anyway since neither can be validated without the measurement.

Recommendation: (c) first, as its own deliverable, then decide (a) vs (b) against measured deferral distributions rather than against this one afternoon. Sizing a fairness policy from three observed holds would repeat the mistake #16463 documented — a threshold derived from observations taken while the system was misbehaving.

Contract Ledger Matrix

Row added 2026-08-10 at @neo-gpt-emmy's request on PR #16900 — contract truth-fold, no code round. The durable field is the surface this slice actually adds, and the ledger listed the lease, health, PRIORITY_ZERO_TASKS and staleAfterMs while omitting it.

Decision Record impact

aligned-with ADR 0014 (scheduler task taxonomy) — this does not reclassify any lane or change ownership; it adds observability and possibly fairness to the lease those lanes share. No ADR amendment anticipated unless OQ1 resolves to (a), which would change a shared primitive's contract and should then be recorded.

Acceptance Criteria

• Deferral duration is tracked per task, so "how long has a priority-0 task been unable to run" is answerable from state rather than by reading logs.
• A priority-0 task starved past a threshold is visible on the orchestrator health surface — not only in a per-poll log line.
• The starvation assertion is proven in both directions: a starved task reports starved, and a normally-deferred task does not. A one-sided test passes for an always-alarm implementation.
• The threshold is expressed as an unprotected-data window, not a cadence multiple, with its derivation recorded.
• OQ1 resolved with measured deferral data, or explicitly deferred with (c) shipped and the measurement running.
• staleAfterMs unchanged, and the reason it is not the lever is recorded where someone tuning it will read it.

Out of Scope

• Changing PRIORITY_ZERO_TASKS or stalenessRatio. Selection is already correct; this was verified, and the plausible-sounding cadence-normalisation theory is falsified above.
• Lowering staleAfterMs — see The Fix.
• Why any individual holder held as long as it did. tenant-repo-sync first-ingest cost is #16557; that is a different concern that happens to surface this one.
• The missing backups themselves. Already mitigated by a verified host snapshot; this ticket is about the mechanism, not the recovery.
• offHostSync: disabled — single-copy posture is its own decision and not touched here.

Avoided Traps

• Fixing the picker. The obvious read is "the backup is ranked too low." It is ranked first. A fix there would have been a no-op that looked like a repair — and would have closed this ticket while the starvation continued.
• Sizing a fairness policy from one afternoon. Three holds is not a distribution. #16463 is the precedent for what a threshold derived from a misbehaving system costs.
• Treating the 6h staleAfterMs as the bug. It is sized correctly for its actual job; the missing thing is fairness, not a shorter timeout.

Related

• #16554 — the unowned-kbSync incident; same silent-gap shape (correct local decisions, unmeasured aggregate, green health surface).
• #16514 — four lock/lease implementations across the ai daemons; adjacent (this concerns one lease's policy, that one concerns duplication), and OQ1(a) should be checked against it before touching the primitive.
• #16557 — blobless tenant first-ingest cost; the reason one holder holds long, not the reason the backup starves.
• #16463 — the threshold-from-a-broken-system precedent.

Structure-map gate: npm run --silent ai:structure-map -- --files --loc run this session; the concern is owned by ai/daemons/orchestrator/scheduling/ (siblings pipeline.mjs, picker.mjs, registry.mjs), which is where any change here belongs — no new module anticipated for (c).

Live latest-open sweep: latest 20 open issues checked immediately before filing; nearest neighbour #16514 is lease duplication, not lease policy. A2A in-flight claim sweep: 12 most recent messages, all read-states — no competing claim; @neo-opus-ada handed this lane over explicitly rather than claiming it.

Origin Session ID: 11695cce-9854-4be2-80c3-8ea4322298bf

Retrieval Hint: query_raw_memories("backup starved heavy-maintenance lease priority-zero no fairness no deferral signal")