LearnNewsExamplesServices
Frontmatter
id16262
titleHeavy-maintenance lease liveness is vacuous for cross-boot pid-1 container epochs
stateClosed
labels
bugaiarchitecture
assigneesneo-kimi-phoebe
createdAtAug 1, 2026, 3:03 PM
updatedAtAug 1, 2026, 4:25 PM
githubUrlhttps://github.com/neomjs/neo/issues/16262
authorneo-kimi-phoebe
commentsCount0
parentIssuenull
subIssues[]
subIssuesCompleted0
subIssuesTotal0
contentTrust
projected
quarantined0
signals[]
blockedBy[]
blocking[]
closedAtAug 1, 2026, 4:25 PM

Heavy-maintenance lease liveness is vacuous for cross-boot pid-1 container epochs

Closed Backlog/active-chunk-11 bugaiarchitecture
neo-kimi-phoebe
neo-kimi-phoebe commented on Aug 1, 2026, 3:03 PM

Context

Surfaced by the #16208 gap-0 window specimen and the intake V-B-A on #16210 (issuecomment-5151446195): the heavy-maintenance lease's process-epoch discriminator — merged in PR #16220 as isLeaseStale's currentProcessStartedAt check — only fires on pid equality. When a lease's recorded pid differs from the reclaiming process's pid, classification falls through to isPidAlive(lease.pid), and pid 1 is vacuously alive inside every container boot (the init process). A dead epoch's pid-1 lease therefore reads live indefinitely across a container recreate: old node ran as pid 1, the recreated container's new init also runs as pid 1, and the lease of a process that no longer exists is honored as if it did.

The window specimen proved the adjacent case in production: the pre-stop wedged heavy-maintenance-lease.json (acquiredAt 11:48:35) survived the 12:03 boot reclaim and blocked the post-restoration backup until the operator approved a manual rm. That instance is closed by the already-merged epoch guard (ff7f624, on dev) once the plane rebuilds past the 28.5h-stale image — docker exec grep -c CURRENT_PROCESS_STARTED_AT on the running container returns 0, confirming the image predates the guard. This ticket covers the gap the guard does not: cross-pid / recreate.

Live latest-open sweep: checked latest 20 open issues at 2026-08-01T13:05Z; no equivalent found (#16210 is closed with its ACs dispositioned; #16258/#16259 are wake-lane, not lease-lane). A2A in-flight claim sweep over the last 30 messages at the same timestamp: the only lease-domain signal is @neo-opus-vega's lane pointer to me — no competing claim.

The Problem

heavyMaintenanceLeasePrimitives.mjs:isLeaseStale discriminates a previous-epoch lease only when Number(lease.pid) === Number(currentPid). The container-recreate path defeats it deterministically:

  1. Container epoch A writes the lease with pid: 1 (its node was the container init).
  2. The container is recreated (mount fix, image rebuild, compose up --force-recreate).
  3. Epoch B's node runs as pid 7 (or any non-1 pid) — or as pid 1 again, in which case the epoch guard fires correctly.
  4. In the non-equal-pid case, the epoch check is skipped and isPidAlive(1) returns true — the new container's init is alive with pid 1, vacuously. The lease from the dead epoch reads as live-held.
  5. Heavy maintenance (backups among them, per the window specimen) defers until staleAfterMs expiry — or indefinitely when the classification path only consults liveness.

The lease payload (buildLeasePayload) currently records owner, reason, metadata, pid, token, acquiredAt, staleAfterMs, expiresAt — nothing that identifies which boot wrote it.

The Architectural Reality

  • heavyMaintenanceLeasePrimitives.mjsisLeaseStale (the discriminator), buildLeasePayload (the payload shape), inspectHeavyMaintenanceLease(Sync) (the classification entry both acquire paths consult).
  • The container's hostname IS its container ID, which changes on recreate — a free, already-true boot discriminator for the exact failure case (recreate), composed from os.hostname() with no config or env read (ADR 0019-clean).
  • The orchestrator already persists a per-boot advisory fact (recordBootIdentityFact / the boot-identity surface) — prior art for "which boot am I" as a recorded value.
  • ai/daemons/shared/fileLease.mjs (merged #16242) solves the same namespace-blindness for the authority lease with heartbeat-TTL. Option (b) below names why this ticket does not start there: the heavy-maintenance lease's staleAfterMs semantics deliberately allow long work under one owner (renew-or-lose), which is a different liveness contract from the authority lease's poll-cadence heartbeat — unification is a deeper alignment, not this fix.
  • Precedent shape: the epoch guard (CURRENT_PROCESS_STARTED_AT) was added to the same module by PR #16220; this extends that exact pattern one field over.

The Fix

(a) Boot-identity discriminator in the lease payload (recommended, this ticket).

  1. buildLeasePayload gains bootId (default os.hostname(); injectable for specs). Additive field — no schemaVersion, matching the receipt-side precedent of #16241's additive integrity field: old payloads simply lack the field.
  2. isLeaseStale gains a boot-identity clause ahead of the liveness probe: lease.bootId present and ≠ the current boot's → stale, regardless of pid liveness. Absent bootId (pre-field payloads) falls through to the existing epoch + liveness path unchanged — no retroactive condemnation.
  3. The epoch guard stays as the same-boot discriminator (pid reuse within one boot); the bootId clause is the cross-boot discriminator (recreate, cross-pid).
  4. Specs: pid-1 lease from a different bootId with isPidAlive forced true → stale (the specimen's generalized form); same-bootId + dead pid → stale via the existing path; same-bootId + live holder → not stale; pre-field payload → unchanged behavior.

(b) Deferred — TTL/heartbeat unification with ai/daemons/shared/fileLease.mjs. The deeper compose Vega named. Not this ticket: the heavy lease's renew-or-lose staleAfterMs contract differs deliberately from poll-cadence heartbeats, and merging the families belongs to its own design pass. @neo-opus-vega holds the ruling on whether (b) ever happens; this ticket's (a) is compatible with either outcome.

Contract Ledger Matrix

Target Surface Source of Authority Proposed Behavior Fallback Docs Evidence
lease payload (buildLeasePayload) heavyMaintenanceLeasePrimitives.mjs additive bootId field, default os.hostname(), injectable pre-field payloads unchanged — no schemaVersion, no rejection module JSDoc spec: payload carries bootId
staleness classification (isLeaseStale) same bootId mismatch with the current boot ⇒ stale, ahead of any pid-liveness probe absent bootId → existing epoch + liveness path, byte-identical module JSDoc spec: cross-boot pid-1 lease reclaims despite vacuous-alive probe
same-boot discrimination CURRENT_PROCESS_STARTED_AT epoch guard (PR #16220) unchanged existing epoch specs stay green

Decision Record impact

aligned-with ADR 0019 (no new env/config read — hostname is a syscall, not a leaf) and aligned-with ADR 0014 (the heavy-maintenance lane's ownership taxonomy is untouched; this is liveness plumbing inside the owning role's own lease). Neither amended.

Acceptance Criteria

  • A lease carrying a bootId different from the current boot's is classified stale even when isPidAlive(lease.pid) returns true — the recreate specimen's generalized form, spec-pinned.
  • A lease with no bootId (pre-field payloads) classifies exactly as before (epoch guard + liveness), spec-pinned.
  • A live same-boot holder is never staled by the new clause (positive control).
  • buildLeasePayload emits bootId by default from os.hostname() and honors an injected value.
  • Post-merge: on the next container recreate with an in-flight heavy-maintenance lease, the new epoch reclaims without manual intervention — recorded as PMV against the rebuilt plane.

Out of Scope

  • The stale-image specimen itself — closed by the rebuild past ff7f624 (no code here).
  • The (b) unification of the two lease families' liveness contracts (named, deferred, Vega's ruling).
  • The heavy-maintenance lifecycle guard (lifecycleGuard.mjs) — orthogonal; it serializes transitions, not staleness.
  • The container rebuild / deploy-authority lane (#16167 cleanup series).

Avoided Traps

  • Editing the epoch guard to fire without pid equality — that would stale a live older holder on the same boot (a legitimate long-lived predecessor within the same container run), trading the recreate gap for a same-boot false kill.
  • A new config leaf for the boot id — ADR 0019 §3's hidden-default class; os.hostname() is already the deployment truth, and a leaf would let config disagree with it.
  • schemaVersion bump on the payload — condemns every pre-field lease on disk; the additive field is the #16241 receipt-side precedent.

Related

  • #16210 — the window specimen + this intake's receipts (issuecomment-5151446195)
  • #16242 / PR #16242 — the merged authority-lease TTL family (compose-not-duplicate boundary)
  • #16208 — the gap-0 window that produced the specimen
  • PR #16220 — the epoch guard this extends
  • #16167 — the migration's cleanup series (rebuild lane owner)

Origin Session ID: f724ffa5-6a4b-430b-b8a1-2cd0e6324caf

Retrieval Hint: heavy maintenance lease boot identity discriminator container recreate pid 1 vacuous liveness stale reclaim

tobiu referenced in commit 0175f6a - "fix(orchestrator): boot-identity discriminator for the heavy-maintenance lease (#16262) (#16266) on Aug 1, 2026, 4:25 PM
tobiu closed this issue on Aug 1, 2026, 4:25 PM