A live heavy-maintenance holder cannot be preempted, and a deployed plane has been starved for 24h

Context

Successor to #16817, which described this mechanism precisely and was closed NOT_PLANNED on 2026-08-09. This ticket does not dispute that judgement — it supplies the fact that was missing when it was made: the condition is now live on a deployed plane, and has been for 24 hours.

Filed at operator direction (@tobiu, 2026-08-19): deployment-readiness work is sequenced behind this, so it is the item that currently blocks recommending the next deployment.

Live latest-open sweep: latest 10 open issues at 2026-08-19T09:0xZ; state:all keyword sweeps for preempt non-terminating holder lease, heavy maintenance watchdog abort, lease holder never yields. #16817 is the only match and it is closed. A2A claim sweep: no in-flight claim.

The Problem

The live instance, read off the affected plane

mc healthcheck, 2026-08-19T08:56Z:

status: degraded
details: [
  "Heavy-maintenance starvation: memory-summary-backfill deferred since 2026-08-18T10:42:28Z,
   message-concept-harvest deferred since 2026-08-18T14:38:01Z,
   provider-residency-repair deferred since 2026-08-18T08:19:10Z,
   summary deferred since 2026-08-18T10:03:31Z
   (lease holder: tenant-repo-sync)"
]

get_deployment_state_snapshot → heavyMaintenance.retry:

lastSuccessAt    : 2026-08-18T12:26:25Z   (20.6 h before observation)
retriesRemaining : 0
nextAttemptAtMs  : 2026-08-19T12:26:20Z   — exactly 24 h after the last success

Four heavy-maintenance tasks deferred 18.3 – 24.6 hours, one lease holder, retry budget exhausted, next attempt on a 24-hour interval. provider-residency-repair has been deferred since one minute after the KB server started.

The deferral survived a restart. mc-server restarted at 2026-08-19T04:51:45Z; the deferral timestamps still read from 2026-08-18. So this does not clear itself on a container bounce, which removes "redeploy and it goes away" as a remedy — and means a deployment would land on an already-degraded plane.

Why the shipped mitigations do not cover it — verified, not assumed

Everything built for this family is already deployed on that plane (deployed revision 2397b940, 2026-08-17):

Every one of these makes the starvation visible. None of them ends it. The observability half of this family is complete and the preemption half does not exist.

The mechanism, re-verified on current dev (ee9c7d1f27)

#16817 is ten days old, so its claims were re-checked rather than cited:

• ai/daemons/orchestrator/services/heavyMaintenanceLeasePrimitives.mjs contains zero occurrences of setTimeout, AbortController, AbortSignal, or Promise.race. There is no timer, no cancellation channel and no watchdog anywhere in the primitives.
• isLeaseStale() (:122) decides staleness from staleAfterMs plus pid liveness (:164, isPidAlive). A holder stuck inside a live call has a live pid, so it is classified active, never stale. #16817's sentence — "A holder stuck inside a live call is not abandoned — it is holding" — is still literally true of the code.
• maxActiveHoldMs is enforced through shouldYield(), a pure predicate the holder must choose to call between batches (#14259). A holder that never reaches a between-batch checkpoint never asks, and nothing asks on its behalf.

So both bounds remain structurally unable to see a holder that is alive and not progressing. Nothing has changed since the NOT_PLANNED decision except the evidence.

The Architectural Reality

• ai/daemons/orchestrator/services/heavyMaintenanceLeasePrimitives.mjs — acquireHeavyMaintenanceLease, isLeaseStale, inspectHeavyMaintenanceLease and their sync overloads. No preemption primitive.
• ai/daemons/orchestrator/services/HeavyMaintenanceLeaseService.mjs:158 — the withHeavyMaintenanceLease(task, {...}) call site; await task(acquisition) inside try/finally, which is where a bound would have to attach.
• ai/daemons/orchestrator/services/MaintenanceBackpressureService.mjs — owns the deferral/backpressure accounting the healthcheck reports from.
• The current holder is tenant-repo-sync, whose own self-throttling behaviour is separately open as #17349 (a partial slice with zero errors accrues a failure streak, so success throttles itself 25×). Whether #17349 is the reason this particular holder is not progressing is unestablished — it is a plausible upstream and it is not evidence. That question belongs to #17349, not here; this ticket is about a holder of any kind being unpreemptable.

The Fix

The shape #16817 already identified, restated with the observation that its cost was the reason for NOT_PLANNED and its absence now has a measured cost:

1. A bound that does not require the holder's cooperation. withHeavyMaintenanceLease is the one seam every heavy task passes through, so an AbortSignal handed to task(acquisition) plus a timer on the wrapper reaches all of them without touching any individual task. Tasks that ignore the signal are no worse off than today; tasks that honour it become preemptable.
2. A holder that has not progressed must become reclaimable. Liveness of the pid is the wrong liveness. A holder that has made no checkpoint in N intervals should read stale even with a live process — which is a change to isLeaseStale, not a new subsystem.
3. Whatever lands must be observable from the snapshot, so the next instance is diagnosable without reading a plane through MCP as this one required.

Deliberately not prescribed: killing the holding process. That is the blunt version, it loses in-flight work, and #16459's 949 orchestrator restarts are the argument against reaching for it.

Acceptance Criteria

• A heavy-maintenance holder that never reaches a checkpoint is preempted, or reclassified reclaimable, within a bounded interval — asserted by a fixture whose task returns a promise that never settles. A task that completes slowly passes under both the old and new implementations and therefore does not count as coverage.
• A holder that does checkpoint keeps its existing cooperative behaviour: shouldYield() semantics unchanged, no reduction in maxActiveHoldMs. Asserted as a negative control so the new bound cannot be a catch-all that preempts healthy holders.
• In-flight work is not destroyed by the reclaim path — the preempted task learns it was preempted rather than being killed mid-write.
• The snapshot reports why a lease was reclaimed and which holder lost it.
• A mutation removing the new bound fails the never-settles fixture and only it.

Out of Scope

• #17349 — whether tenant-repo-sync self-throttles into a non-progressing state. Upstream, separately open, and this fix must work regardless of which holder is stuck.
• The four starved tasks' own behaviour. They are the victims, not the defect.
• Off-host backup durability — that plane is posture: "opted-out" by deliberate configuration.
• Clearing the current live instance. #17352 (an operator can clear tenant-sync backoff without restarting) is already merged and is the operational lever for the plane as it stands today.

Avoided Traps

Reopening #16817. It was closed NOT_PLANNED as a judgement, and reopening would silently overturn a decision rather than answering it. A successor carrying the new evidence lets the original decision stand as what it was — correct on the information available then.

Filing this as "the fairness bound is broken". It is not: maxActiveHoldMs does what it was specified to do. The gap is that a cooperative bound cannot bind an uncooperative holder, and calling the bound broken would send the next reader to tune a number.

Treating the deployment as the remedy. The deferral survived a container restart, and none of the 15 ai/ commits pending for that plane touch lease preemption. Deploying would move that plane onto a newer revision that is degraded for the same reason on day one.

Related

• #16817 — CLOSED NOT_PLANNED; described this mechanism first. This is its successor, not its reopening.
• #16566 — OPEN; records the original 13-hour instance where a kbSync re-embed held the slot.
• #17349 — OPEN; the current holder's possible self-throttling upstream.
• #16561 / #17049 / #17295 / #16564 — CLOSED; the observability half, all deployed and all working.
• #14259 — CLOSED; added maxActiveHoldMs + shouldYield(), the cooperative bound this cannot rely on.
• #16822 — CLOSED; kbSync checkpointing 33× later than the bound it must respect.
• #16459 — CLOSED; 949 orchestrator restarts, the argument against a kill-the-holder remedy.

Origin Session ID: fb387768-e68f-4a71-9b6a-3cf9ad4a9e7e

Retrieval Hint: heavy-maintenance lease live holder unpreemptable; pid liveness is the wrong liveness; affected plane starved 24h behind tenant-repo-sync

— Vega (Claude Opus 5, Claude Code) 🌿