LearnNewsExamplesServices
Frontmatter
id17398
titleThe container tenant sweep never consults the lease it holds
stateClosed
labels
bugaiagent-os
assigneesneo-opus-vega
createdAtAug 19, 2026, 10:11 PM
updatedAtAug 21, 2026, 10:03 AM
githubUrlhttps://github.com/neomjs/neo/issues/17398
authorneo-opus-vega
commentsCount0
parentIssue17380
subIssues[]
subIssuesCompleted0
subIssuesTotal0
contentTrust
projected
quarantined0
signals[]
blockedBy[]
blocking[ ] 17414 A lease-caused yield cannot release the lease, because the sweep never learns which cause stopped it
closedAtAug 21, 2026, 10:03 AM

The container tenant sweep never consults the lease it holds

Closed Backlog/active-chunk-17 bugaiagent-os
neo-opus-vega
neo-opus-vega commented on Aug 19, 2026, 10:11 PM

Context

Split from #17380, which enumerates the class ("which heavy tasks consult the lease yield predicate"). This leaf is the one holder the enumeration got wrong twice and the one the live starvation names.

#17047 gave ai/scripts/maintenance/syncTenantRepos.mjs the outer deployment-wide heavy-maintenance lease, closing a real bypass: before it, the container one-shot called TenantRepoSyncService.runTask() directly and took only the service's narrower tenant-repo-sync-lease.json. That fix was correct. Nothing gave the new holder a way to ask how long it had held.

Live reading, 2026-08-19 ~19:50Z. Five waiters past the one-hour degrade threshold, every breach naming holder tenant-repo-sync: summary 33.8h, memory-summary-backfill 33.1h, message-concept-harvest 29.2h, graphlog-compaction 11.2h, provider-residency-repair 4.0h. The server's own watchdog states the verdict: "the fairness yield bound has been exceeded; the lease pipeline is not admitting its waiters."

Separating observation from inference: the breach list, durations, holder string and watchdog text are observed. Which shape the holder's occupancy has — one long hold versus continuous re-acquisition — is not established, and this ticket does not claim it (see Out of Scope).

The Problem

The owner string identifies the holder, and it is not the file #17380 pointed at. TenantRepoSyncService stamps `tenant-repo-sync:${'manual'|'scheduler'}` (:1468); the CLI stamps the bare 'tenant-repo-sync' (syncTenantRepos.mjs:207). The live watchdog reports exactly 'tenant-repo-sync', so the shared-lease holder is the CLI path. #17380's first draft named the pure scheduling module; its first correction named the service. Both were wrong about which file holds the shared lease.

The holder consults nothing. syncTenantRepos.mjs contains zero shouldYield references. It acquires resolveHeavyMaintenanceLeasePath(...) — the shared lease — and runs the whole sweep beneath it.

The bound that does exist measures the wrong thing. Inside, the sweep passes shouldYield: createSliceBudgetPredicate({startedMs, sliceBudgetMs}) per repo. That predicate is () => now() - startedMs >= sliceBudgetMs — pure elapsed time against this repo's admission, with no lease term. Its own docblock is explicit that the per-repo anchoring is deliberate: a sweep-wide budget would be spent by the first admitted repo and starve the tail.

So the two bounds answer different questions, and the gap between them is the defect. The slice budget is fairness between repos; the lease bound is fairness between tasks. A sweep over N repos honours every per-repo budget and still occupies the exclusive-heavy slot for roughly N × sliceBudgetMs, because the slice budget rotates within the sweep rather than ending it. Every budget the holder can see is satisfied while waiters starve for a day and a half.

The precedent exists and was not followed here. #16823 wired kbSync by building a predicate over the acquisition and consulting it per provider chunk. syncKnowledgeBase.mjs is the only one of nine lease-acquiring paths that does.

The Architectural Reality

  • ai/scripts/maintenance/syncTenantRepos.mjs:203-212 — the withLease wrapper: shared leasePath, owner: 'tenant-repo-sync', reason: 'container-one-shot'. :166-170's docblock states the two-lease topology.
  • ai/daemons/orchestrator/scheduling/tenantRepoSync.mjs:111-113createSliceBudgetPredicate, pure elapsed time, no lease term.
  • ai/daemons/orchestrator/services/TenantRepoSyncService.mjs:1663syncTenantRepos, which owns sliceBudgetMs and the per-repo ingest call. runTask (:1412) is a different scope, which is why a parameter declared only there is not visible at the ingest call.
  • ai/daemons/orchestrator/services/HeavyMaintenanceLeaseService.mjs:144shouldYield(lease), the thin wrapper over the pure primitive with maxActiveHoldMs injected.
  • ai/scripts/maintenance/syncKnowledgeBase.mjs:41-49 — the kbSync precedent, including the config trap: maxActiveHoldMs lives on orchestrator.heavyMaintenance, not the sibling orchestrator.heavyMaintenanceLease.
  • ai/configBase.mjs:1780maxActiveHoldMs: leaf(HOUR_MS / 2, …); 30 minutes, live, not overridden on any plane checked.
  • ai/daemons/orchestrator/scheduling/pipeline.mjs:787leaseHolder derives from inspection.lease?.owner, which is why the owner string is the discriminator.

The Fix

AMENDED 2026-08-20, same drift as the ledger below. Items 1 and 2 previously read "in syncTenantRepos.mjs" (the CLI-only premise) and "yield if EITHER bound is reached" (the boolean OR) — both refuted. @neo-gpt-emmy's Required Action named the Contract Ledger; this section carried the identical staleness one heading higher and was not named. Fixing the class rather than the cited instance, because the next reader would have hit whichever copy they reached first.

  1. Build the outer lease's fairness vote once, in a config-aware module both holders already importcreateLeaseYieldVoter in services/HeavyMaintenanceLeaseService.mjs, from the acquisition descriptor each holder already has. One home for the config branch, so the sibling-leaf trap is spelled once rather than per holder.
  2. Both outer acquisitions supply it — the scheduler's through taskOptions (beside the inherited token and the release callback it already derives from the same acquisition), the CLI's from the descriptor withHeavyMaintenanceLease has always handed its task. Neither is the live holder by assumption.
  3. Compose to a CAUSE, not a booleancreateYieldCauseResolver returns the highest-precedence firing cause or null. Every lease-exit step is conditional on the cause, so a resolver answering only "something said stop" makes the exit contract unexpressible. A caller holding no outer lease passes nothing, and the boolean projection the sweep consumes is then the slice predicate's own answer, byte-for-byte — which is every in-process scheduler path.
  4. Give the clear-backoff branch no vote. It is a short manifest rewrite that must finish atomically; a yield there invites a half-applied clear.
  5. Declare the parameter in the scope that consumes it (syncTenantRepos, beside sliceBudgetMs) and thread it from runTask — declared one frame too high, it is an out-of-scope reference inside the per-repo try, which the repo-level catch converts into an ordinary repo failure.

Contract Ledger Matrix

AMENDED 2026-08-20 to the shipped surface, on @neo-gpt-emmy's Required Action against PR #17424. The table below previously specified composeYieldPredicates as a boolean OR, a leaseShouldYield parameter, and a CLI-local buildLeaseYieldPredicate — i.e. the exact contract the Drop+Supersede refuted. The premise section and ACs above were rewritten hours earlier and this table was not, which made the ticket's authoritative surface still teach the discarded API. Recording that rather than silently overwriting it: a corrected premise with one stale downstream section is the same defect as a corrected premise with stale ACs, and I had already been caught by the ACs version of it on this same ticket.

Target Surface Source of Authority Proposed Behavior Fallback Docs Evidence
YIELD_CAUSE_LEASE / YIELD_CAUSE_SLICE / YIELD_CAUSES (new, pure) scheduling/tenantRepoSync.mjs — the pure module owns the vocabulary frozen, precedence-ordered cause names; lease outranks slice the voter is built in a config-aware module while the vocabulary lives here, so two spellings of one cause would be rejected only once a deployment reached the bound
createYieldCauseResolver(voters) (new, pure) scheduling/tenantRepoSync.mjs (sibling of the slice predicate) returns () => String|null — the highest-precedence firing cause, never a boolean. Precedence by declared order, not argument order. Unknown cause ⇒ TypeError. Non-function vote ⇒ dropped. A throwing voter propagates no voters ⇒ null a boolean makes every lease-exit step unexpressible; that shape error is why a holder that has starved 5 waiters since Aug 18 keeps rotating
createLeaseYieldVoter(acquisition) (new, config-aware) orchestrator.heavyMaintenance.maxActiveHoldMs, read at vote time in services/HeavyMaintenanceLeaseService.mjs returns {cause: 'lease', vote} from any acquisition descriptor — the single home for this reading, shared by both outer holders no acquisition / no leasenull, never a voter that always answers false the sibling orchestrator.heavyMaintenanceLease leaf holds only staleAfterMs; reading it resolves undefined, a falsy bound never votes, and the no-op is indistinguishable from a wired predicate
MaintenanceBackpressureService taskOptions.leaseYieldVoter the scheduler's own acquisition (acquireLeaseFn) third value derived from that acquisition, beside the inherited token and the release callback compatible-pair bypass (no own token) ⇒ null — that candidate holds no lease to yield the runner it dispatches to previously dropped taskOptions entirely
pipeline.mjs 'tenant-repo-sync' runner the taskOptions acquireLeaseAndExecute already passes signature widened to (taskName, reason, onSuccess, taskOptions); forwards leaseYieldVoter to runTask absent taskOptions (non-heavy short-circuit path) ⇒ null, never undefined the scheduler's hold could not be voted against at all while this edge dropped its fourth argument
TenantRepoSyncService runTask / syncTenantRepos leaseYieldVoter whichever acquisition dispatched the run declared on both frames — syncTenantRepos is where the per-repo vote is composed, so a parameter declared only on runTask is an out-of-scope reference inside the per-repo try null ⇒ slice budget alone, byte-identical to pre-change cloud-deployment docs 9 lease paths, 1 consulting the bound before this
syncTenantRepos.mjs CLI invoke the descriptor withHeavyMaintenanceLease hands its task the closure now takes that descriptor and builds the voter from it --clear-backoff branch receives no voter — an atomic manifest rewrite must not yield partway a half-applied clear is worse than holding the lease past its bound
the sweep's per-repo shouldYield the composed resolver the resolver's boolean projection (cause !== null), so the cause is readable without any exit acting on it with a null lease voter the projection is the slice predicate's own answer consumption — tail-admission stop, cohort commit, outer release — is #17414

Decision Record impact

aligned-with ADR 0022 — cooperative yield only. No hard preemption, no leaseMonitor.mjs wiring, which ADR 0022 anti-anchors and #17379 died proposing.

⛔ PREMISE CORRECTED 2026-08-20 — this ticket's owner discriminator was FALSE

PR #17399 was closed unmerged under a Drop+Supersede from @neo-gpt-emmy (review). The starvation this ticket describes is real and ongoing. The prescription below was wrong in two ways, and both were authored here rather than introduced by the implementation.

1. There are TWO outer lease holders, not one

This ticket named the CLI as the outer acquisition. MaintenanceBackpressureService.mjs:992-998 is a second outer acquisition — the scheduler's — and it is indistinguishable from the CLI at the point where the vote is cast. A repo-level vote wired into one of them cannot terminate a hold the other owns. Falsifier coordinates: scheduler acquisition MaintenanceBackpressureService.mjs:992-998, scheduler runner pipeline.mjs:452-460, watchdog owner projection pipeline.mjs:786-787, per-repo vote TenantRepoSyncService.mjs:2401-2417, all-repo admission TenantRepoSyncService.mjs:2922-2923.

2. A boolean OR over yield voters erases the cause the exit contract needs

The implemented composeYieldPredicates returned voters.some(v => v() === true) — which is the wrong shape, not a wrong detail. A lease yield and a slice yield require different exits:

yield cause required exit
slice budget rotate to the next repo; the lease is still ours
outer lease age stop admitting the tail, persist the active cohort's resumable state, release the outer lease

A vote that answers only true/false cannot express either branch, because all three lease-exit steps are conditional on the cause the OR discarded. Any successor must carry a lease-specific yield outcome through sweep admission, not a boolean.

Corrected prescription

  • Both outer-acquisition paths carry the acquisition-age vote — scheduler and CLI, with neither hardcoded as the live holder.
  • A lease-bound yield stops further tail admission after the active cohort, commits resumable state, and releases the outer lease.
  • A slice-only yield continues rotating repos, unchanged. Preserve this as an independent control so the lease-bound exit cannot silently absorb it.
  • Every production propagation edge is mutation-guarded. Stub-only dispatch evidence does not close this: the prior implementation's tests traversed a stub, not the production edges.

SPLIT, 2026-08-20. The reviewer's prescription names two obligations and they are not one PR:

leaf owns order
this ticket both outer acquisitions vote, and the verdict carries its cause (lease vs slice) rather than a boolean first — nothing downstream is expressible without the cause
#17414 the lease-caused exit contract: stop tail admission after the active cohort, commit resumable state, release the outer lease second, blocked_by this ticket

So this ticket's scope narrows to producing a cause-bearing verdict from both outer-acquisition paths. Consuming it — the three exit steps — moved to #17414, which is where the slice-cause control also lives so the lease exit cannot silently absorb slice behaviour.

Salvage — retained, do not rebuild

The correct AiConfig branch (orchestrator.heavyMaintenance.maxActiveHoldMs, not the sibling heavyMaintenanceLease — reading that one yields undefined, and a falsy bound never votes); the acquisition-age predicate; the no-lease fallback returning null rather than an always-false predicate; the clear-backoff exclusion; and the option-threading pieces.

Acceptance Criteria

Rewritten 2026-08-20 with the premise. The prior list described the refuted shape — it named the CLI as the outer acquisition and asserted on "removing the lease voter from the composition", i.e. the boolean OR that made the exit contract unexpressible. Implementing to it would rebuild the dropped design.

  • RED-PROOF, scheduler path: against dev, an assertion that the scheduler's outer acquisition (MaintenanceBackpressureService.mjs:992-998, runner pipeline.mjs:452-460) contributes an acquisition-age vote must FAIL. This is the arm that catches the original premise error, because the refuted implementation wired the CLI only and was thoroughly tested that way.
  • RED-PROOF, CLI path: the same assertion for the CLI acquisition must FAIL against dev. Both arms are required — one passing arm is exactly the evidence that produced a false owner discriminator.
  • The verdict carries its CAUSE, and a boolean cannot satisfy it. An assertion that a lease-caused yield is distinguishable from a slice-caused yield must fail against dev. A fixture asserting only that "something voted to yield" passes before and after and proves nothing — that is the shape error, restated as an arm.
  • Type-level guard against re-flattening: a fixture asserts the verdict is not a boolean, so a later refactor cannot silently collapse two causes back into one bit. The refuted code read as the elegant option; nothing in the suite objected.
  • Negative control: a holder inside the bound produces no lease cause, asserted on the real predicate rather than a stub, so a changed comparison operator is caught.
  • No-lease equivalence: a caller with no acquisition yields exactly the pre-change slice-budget-only behaviour, asserted against the raw predicate's own answer rather than against a mock.
  • Mutation guard, per path: removing the scheduler voter turns only the scheduler arm red; removing the CLI voter turns only the CLI arm red. Per-arm specificity is the requirement — a suite that goes red on any edit is sensitive, not specific, and a reviewer had to teach me that distinction three times on #17336 before an arm actually isolated its own call site.
  • A throwing voter propagates rather than reading as "do not yield": swallowing converts a broken bound into starvation that looks healthy, which is the failure this whole lane exists to end.
  • The clear-backoff branch receives no vote.
  • Consumption is out of scope and asserted to be absent here. This leaf makes the cause readable; #17414 makes it acted on. An arm confirms this leaf changes no exit behaviour, so the two can be reviewed independently.

Out of Scope

  • Which occupancy shape the live instance has. starvedForMs measures the waiter's wait, so continuous re-acquisition and a single long hold are indistinguishable from it. #17380 carries that open question.
  • Observing a starved peer actually interleave (#17380 AC-3) and mechanical detection of heavy tasks that consult nothing (#17380 AC-5). Both stay on the parent; neither is decidable from this leaf's unit surface.
  • The other seven non-consulting lease paths. Enumerated on #17380; wiring them is not this leaf.
  • Hard preemption, leaseMonitor.mjs, any lease-side change — ADR 0022.
  • The holder's own throughput (#17349) and clearing the current live instance (#17352, merged, the operational lever).

Avoided Traps

  • Fixing the file the parent ticket named. #17380 pointed first at the pure scheduling module, then at the service. The shared-lease holder is a third file, and the owner string is what proves it — tenant-repo-sync bare versus tenant-repo-sync:scheduler.
  • Searching for one spelling of the call. The parent's enumeration searched withHeavyMaintenanceLease( and missed four paths that call it as withLease(, including this holder. Its control survived that stage, so the sweep read as complete.
  • Replacing the slice budget instead of composing with it. The per-repo anchoring is deliberate and documented; a sweep-wide budget starves the tail it serves.
  • Reading the sibling config leaf. maxActiveHoldMs is on orchestrator.heavyMaintenance; the adjacent heavyMaintenanceLease holds only staleAfterMs. Reading the wrong one yields undefined, and a falsy bound means the primitive never votes — a no-op that looks exactly like a wired predicate at every call site.
  • Declaring the parameter one frame too high. Declared on runTask alone it is an out-of-scope reference inside the per-repo try, which the repo-level catch converts into an ordinary repo failure: a checkpoint that stays null while the sweep reports no cause. Caught by an existing fixture, not by reading.

Related

  • #17380 — parent; the class enumeration and the two ACs that stay there.
  • #17047 — gave this holder its outer lease; this is its direct successor.
  • #15763 — added the service's narrower lease, the inner half of the two-lease topology.
  • #16823 — the kbSync precedent this mirrors.
  • #16561 — a prior starvation instance (priority-0 backup starved 8.5h).
  • ADR 0022 — heavy-maintenance scheduling fairness; cooperative yield only.

Origin Session ID: 8cbd588b-be06-4a56-9997-1058f2a3a07b

Retrieval Hint: query_raw_memories("container tenant sweep holds shared heavy maintenance lease without consulting maxActiveHoldMs; createYieldCauseResolver and createLeaseYieldVoter replaced the refuted composeYieldPredicates boolean OR; slice budget rotates within the sweep") — the refuted name is kept as a search anchor on purpose, since the archaeology of this lane is why the shipped shape looks the way it does.

tobiu referenced in commit 2e8cbb8 - "feat(ai): the yield verdict reports which bound fired, not that one did (#17398) on Aug 21, 2026, 2:31 AM
tobiu referenced in commit dcfde12 - "feat(ai): both outer lease holders vote, and the vote reaches the sweep (#17398) on Aug 21, 2026, 2:31 AM
tobiu referenced in commit d377507 - "test(ai): the consumption-absent boundary crosses production instead of a fixture (#17398) on Aug 21, 2026, 2:31 AM
tobiu referenced in commit dce6947 - "test(ai): the tail-admission arm gets a repo outside the active cohort (#17398) on Aug 21, 2026, 2:31 AM
tobiu referenced in commit b81884a - "refactor(ai): describe behaviour instead of recording how the fix was found (#17398) on Aug 21, 2026, 2:31 AM
tobiu closed this issue on Aug 21, 2026, 10:03 AM