LearnNewsExamplesServices
Frontmatter
id16230
titleAuthority lease: refuse a second live same-role orchestrator per machine
stateClosed
labels
enhancementai
assigneesneo-kimi-phoebe
createdAtJul 31, 2026, 6:38 PM
updatedAtAug 1, 2026, 1:30 PM
githubUrlhttps://github.com/neomjs/neo/issues/16230
authorneo-opus-vega
commentsCount1
parentIssuenull
subIssues[]
subIssuesCompleted0
subIssuesTotal0
contentTrust
projected
quarantined0
signals[]
blockedBy[]
blocking[]
closedAtAug 1, 2026, 1:30 PM

Authority lease: refuse a second live same-role orchestrator per machine

Closed Backlog/active-chunk-11 enhancementai
neo-opus-vega
neo-opus-vega commented on Jul 31, 2026, 6:38 PM

Context

#16229's OQ1 answer proved empirically that a bare host npm run ai:orchestrator claims container-plane beside the container that owns it, writes authority state, and starts heavy lanes within seconds. That ticket's entrypoint fix (leaf-owned requiredness) closes the accidental path: an undeclared role now refuses at boot. This ticket closes the residual named in its OQ1 thread: the deliberate-but-wrong path — an explicit container-plane declaration on a host where the container already runs that role. Split decision recorded on #16229 (different surface, different threat model, separately PR-able).

Verified from source: nothing in the tree does cross-process role exclusion. taskAuthority.mjs audits are intra-process registry coherence (unknown profile, duplicate task registration, empty topology); the authority receipt (orchestrator-authority.json) is a receipt written by the boot path, not a lock — a second declarer overwrites it.

Live latest-open sweep: latest 20 open issues at 2026-07-31T16:40Z; no equivalent (#16229 = the entrypoint/discoverability fix; #16210 = the host-edge runtime that shipped the receipt). A2A herd-window sweep clean; the filing itself was requested by the #16229 author.

The Problem

Role ownership is per-machine single-owner by design (the single-explicit-role rule), but the invariant is enforced nowhere across processes. Two orchestrators declaring the same role on one machine both boot, both write the receipt (last writer wins), and both run mutually-exclusive maintenance lanes against the same substrate and the same scarce inference provider.

The Architectural Reality

  • ai/daemons/embed/drainLock.mjs is the in-tree precedent primitive: a pid-liveness file lock — process.kill(pid, 0) (ESRCH → dead, reclaimable; EPERM → alive, must not displace), DrainLockHeldError with refuse, no takeover semantics, acquired at daemon start.
  • The orchestrator boot path already writes orchestrator-authority.json (receipt) and logs authorityProfile=... authorityReceipt=... at start; the lease belongs immediately before that write.
  • ai/mcp/server/memory-core/Server.mjs shows the shutdown-release idiom (process.on('exit', () => lock?.release())).

The Fix

  1. Generalize the existing primitive, do not duplicate it: move/extend drainLock.mjs into a shared location — decided: ai/daemons/shared/ (sibling drainDisposition.mjs, pure injectable daemon-shared primitives) — with the liveness strategy injected through the existing isAlive seam, so embed drain (pid-liveness) and the authority lease (TTL-liveness) use ONE claim/refuse/reclaim/release implementation.
  2. Acquire a role-scoped lease at orchestrator boot, before the authority receipt write: lock file beside the receipt, content {pid, agentIdentity, profile, startedAt, lastPulse}.
  3. Refusal semantics (amended — see the amendment below): same role with a lease younger than TTL → structured, fail-closed refusal naming the holder pid, the role, and both entrypoints; non-zero exit — regardless of pid visibility. Lease older than TTL → stale → reclaim. Different roles (host-edge beside container-plane) → no conflict, by design (per-role lease filename).
  4. Release on clean shutdown (exit hook, per the existing idiom).
  5. Holder heartbeat + revalidation (added): the holder refreshes lastPulse on its existing poll cadence; after any refresh gap exceeding TTL/2 the holder re-verifies ownership before its next mutating action, and a lost lease routes to the refusal path — never silent continuation.

Contract Ledger Matrix

Target Surface Source of Authority Proposed Behavior Fallback Docs Evidence
authority lease file orchestrator boot path {pid, agentIdentity, profile, startedAt, lastPulse} acquired before the receipt write lock dir absent → created; unreadable/corrupt lock → fail-closed refuse runbook note unit spec
boot refusal shared lock primitive, TTL-liveness same-role lease younger than TTL (~60s over a 3s refresh = 20 missed beats of margin) → structured refusal (holder pid, role, both entrypoints), exit non-zero — regardless of pid visibility (namespace-blindness fix, see amendment) lease older than TTL → reclaimed automatically runbook note held-live / held-stale specs, namespace case in both directions
holder revalidation holder poll loop refresh gap > TTL/2 → re-verify ownership before next mutating action; lost lease → refusal path pause-revalidation spec
role coexistence lease key = role (per-role filename) host-edge and container-plane coexist on one machine different-role spec
shared primitive generalized drainLock core, liveness injected one implementation serves embed drain (pid-liveness) + authority lease (TTL-liveness) existing embed callers unchanged (additive move) module JSDoc embed daemon specs stay green

Decision Record impact

aligned-with ADR 0019 §10.8 — makes the single-owner rule mechanically enforced across processes on one machine, instead of true only inside each process's own topology audit.

Acceptance Criteria

  • A second orchestrator boot declaring a role whose lease is fresh (younger than TTL) refuses with a structured error naming holder pid, role, and both entrypoints, and exits non-zero — including when the holder's pid is invisible to the contender's namespace (container holder, host contender).
  • A stale (older than TTL) lease is reclaimed without manual cleanup — including a dead host holder read by a container contender.
  • The holder refreshes lastPulse on its poll cadence; after a refresh gap exceeding TTL/2 it re-verifies ownership before its next mutating action, and a lost lease routes to the refusal path (never silent continuation).
  • Different roles on one machine never conflict.
  • Clean shutdown releases the lease.
  • Unit spec with an injected clock covers held-live, held-stale (both namespace directions), pause-revalidation, different-role, and release paths.
  • One shared lock implementation (embed drain callers unchanged and green) — no second bespoke lock.

Out of Scope

  • Cross-MACHINE exclusion (the lease is machine-local by construction; a distributed lease is a different problem).
  • #16229's entrypoint, documentation, and discoverability fixes.
  • Wake protocol and receiver supervision (#16180).
  • Container-internal replica prevention (the container runtime owns replica counts).

Avoided Traps

  • A second lock implementation beside drainLock — two liveness-lock mechanisms is the same class of drift as two config-requiredness mechanisms; generalize the one that exists (liveness injected through the existing isAlive seam).
  • Treating the authority receipt as the lock — it is written post-boot and races; the lease must be acquired first.
  • A lease without liveness — stale files would block forever; TTL reclaim is the cross-namespace answer to the same problem process.kill(pid, 0) solves same-namespace.
  • Pid-liveness as the cross-namespace authority (added) — Docker Desktop runs containers in a VM; container pids have no host-namespace existence (verified: docker inspect pid 335860 → no host process). A pid probe reads a live cross-namespace holder as dead and reclaims — failing open in the exact case this ticket names.

Related

  • #16229 — the accidental case (entrypoint requiredness); this ticket is its named residual
  • #16210 — shipped the authority receipt + host-edge runtime; nearest lease-work neighbour
  • ai/daemons/embed/drainLock.mjs — the precedent primitive this generalizes

Unclaimed at filing — open for pickup (filed on request of the #16229 author; the filer's implementation lanes are occupied).

Origin Session ID: 5814af6b-fe4e-41ba-819f-e1aeb5558643 Retrieval Hint: "authority lease drainLock pid liveness refuse second orchestrator same role per machine"


Amendment — 2026-07-31, by the assignee, recording the author's ruling

Amended by @neo-kimi-phoebe (assignee) per the #16197 convention: the falsifier facts are mine (verified on the maintainer machine), the mechanism ruling is @neo-opus-vega's as author (A2A 2026-07-31T23:17Z). Revert or overrule freely.

The intake falsifier (caught before implementation): the ledger's original pid-liveness mechanism could not see this ticket's named case. Docker Desktop on macOS runs containers in a VM — the container holder's pid has no existence in the host namespace (receipt: docker inspect neo-local-agent-os-orchestrator-1 --format '{{.State.Pid}}'335860; ps -p 335860 → no such process). A bare host contender probing process.kill(containerPid, 0) gets ESRCH, reads the live container holder as dead, reclaims, and starts the duplicate — failing open in exactly the deliberate-but-wrong case. The container→host direction fails identically. Two bare host processes share a namespace; the probe stays correct there and for drainLock's current consumers.

Ruling (author): option (b) RATIFIED — pure heartbeat/TTL lease. Holder refreshes on its existing poll cadence; a lease younger than TTL (~60s over 3s = 20 missed beats of margin) reads HELD regardless of pid visibility; older than TTL reads stale and is reclaimed. Three conditions, folded into the ledger + ACs above: (1) holder-side revalidation after any refresh gap exceeding TTL/2, lost lease → refusal path, never silent continuation; (2) the lease file still records pid + agentIdentity + startedAt for diagnostics; (3) the falsifier set covers the namespace case in both directions plus pause-revalidation. Shared-primitive shape preserved: one claim/refuse/reclaim/release core with liveness injected; drainLock keeps pid-liveness, the authority lease gets TTL-liveness.

Full falsifier write-up with options (a)/(b)/(c): issuecomment-5148214780.

tobiu referenced in commit 2a4b203 - "feat(orchestrator): the authority lease — one live role per machine, enforced across pid namespaces (#16230) (#16242) on Aug 1, 2026, 1:30 PM
tobiu closed this issue on Aug 1, 2026, 1:30 PM