The container-plane orchestrator OOMs and has restarted 949 times

Context

Found 2026-08-03T18:15Z on the maintainer plane while dry-running an unattended update caller (#16454 item 1). The caller refused to proceed with revision-unknown because it could not read the orchestrator's /app/.neo-revision. The reason it could not read it is that the container is almost never running.

RestartCount=949   State=restarting   Restarting=true

The container-plane orchestrator has died and restarted 949 times. Everything the orchestrator owns — scheduled maintenance lanes, Dream, summaries, tenant-repo sync, heal actuation — has been running at whatever duty cycle survives that.

The Problem

It is out of memory, and the lease refusal that shows up first in the logs is a symptom of the restart loop rather than its cause.

The visible error on a docker logs tail is this, repeating every ~15s:

[Orchestrator] Failed to start: FileLeaseHeldError: authority lease
.authority-lease-container-plane is held by orchestrator@21ccb536bbb9 pid 1
(since 2026-08-03T18:09:24Z); orchestrator@21ccb536bbb9 pid 1 refuses to start
a duplicate (single-owner invariant) ... stop the duplicate.

That is correct behaviour being reported misleadingly. The lease descriptor shows a holder that genuinely acquired and pulsed it:

{"pid":1,"owner":"orchestrator@21ccb536bbb9","profile":"container-plane",
 "startedAt":"2026-08-03T18:09:24.177Z","lastPulse":"2026-08-03T18:11:53.465Z"}

startedAt → lastPulse is 149s of live pulsing. So the orchestrator does start, does take the lease, and then dies; the next restart lands inside the 60s AUTHORITY_LEASE_TTL_MS window and is refused as a duplicate of its own dead predecessor. The lease is doing exactly what #16230 built it to do.

Filtering the lease noise out of the log leaves the actual terminal event:

FATAL ERROR: Ineffective mark-compacts near heap limit — JavaScript heap out of memory
FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
  222742 ms: Mark-Compact 501.9 (531.2) -> 474.7 (519.7) MB  (average mu = 0.324) allocation failure
  198365 ms: Mark-Compact 483.6 (530.0) -> 467.6 (513.9) MB  (average mu = 0.199) allocation failure
  161253 ms: Mark-Compact 502.4 (532.0) -> 474.7 (520.4) MB  (average mu = 0.263) allocation failure

It dies at ~500 MB heap, after ~160–220 s of uptime, with the GC mark-utilisation collapsing toward zero — the shape of a process spending its last seconds in back-to-back compaction rather than one that hit a single large allocation.

The ceiling is not the container's. Measured:

So the process is dying at roughly Node's own default old-space cap while the container it runs in has headroom it never uses. Whatever the memory growth is, the runtime is not configured to use the box it was given, and the failure surfaces as a crash loop rather than as pressure.

The Architectural Reality

• ai/daemons/orchestrator/daemon.mjs — the entrypoint; startOrchestrator() acquires the authority lease at :319.
• ai/daemons/orchestrator/authorityLease.mjs — AUTHORITY_LEASE_TTL_MS = 60_000; liveness is TTL, never pid (:12), deliberately, because a container holder's pid has no host-namespace existence. That decision is correct and is not implicated here.
• ai/daemons/shared/fileLease.mjs — the lease primitive; FileLeaseHeldError carries the stop the duplicate remediation string that misdirects in this scenario.
• ai/deploy/docker-compose.yml:47 — the orchestrator's 2g limit, which the running container does not reflect (it has 1 GiB), so the effective limit is coming from elsewhere in the merged config.
• ai/scripts/lifecycle/auto-update/autoUpdateRunner.mjs — how this was found; it refuses to update a plane whose revision it cannot read, which is what turned a silent crash loop into a visible one.

The Fix

Two separable pieces; the second is the real one.

1. Give the runtime the box it already has. No --max-old-space-size is set, so the process caps out well below the container limit. Raising it stops this crash loop — but on its own it converts a fast loop into a slower one if the growth is unbounded, so it is mitigation, not resolution.
2. Establish whether ~500 MB is a leak or a legitimate working set. 949 restarts with a ~3-minute survival time is a strong, cheap reproducer: attach a heap snapshot across two of those cycles and compare retained sets. If it is a leak, this ticket's real deliverable is that fix; if the working set is genuinely that large, the limit and the runtime flag both need to be stated deliberately rather than inherited.

Additionally, and cheaply: the FileLeaseHeldError remediation is actively misleading during a restart loop. "stop the duplicate" sends an operator hunting for a second orchestrator that does not exist. When the refused requester's identity matches the recorded holder's, the message should say so — that it is very likely the requester's own predecessor inside the TTL window, and that the thing to investigate is why the previous instance died.

Contract Ledger Matrix

Decision Record impact

none — a runtime-configuration defect plus a diagnostics correction. No change to the authority-lease contract, whose TTL-not-pid liveness decision (#16230) is correct and is not the cause here.

Acceptance Criteria

• Heap ceilings are per process and explicit — the parent's is scoped to its own invocation, and supervised children receive their own rather than inheriting the parent's.
• Raising the container limit cannot silently raise an unbounded child's implicit ceiling: no supervised child is spawned without an explicit ceiling.
• The container limit is stated against an aggregate budget naming the maximum concurrent Node processes, not against a single heap.
• A task that declares its own ceiling is never overridden by the default.
• An executable assertion pins the parent/child environment boundary.
• The child ceiling resolves through an AiConfig leaf — never a direct env read in the consuming service — and the deployment override has a production writer, not only a reader.
• An invalid override fails closed rather than falling back; Node must never receive a rejected flag and continue with a heap larger than the cgroup.
• A FileLeaseHeldError whose holder identity equals the requester's reports the measured identity match — not a causal claim the frame cannot establish — and does not emit "stop the duplicate".
• The authority lease's TTL-not-pid liveness is unchanged.

Residual transferred to #16463. Recreate-survival and the bounded-vs-leak verdict are L4 — they need a running orchestrator and cannot be proven on a branch. They are not deferred inside this ticket, because a PR must not auto-close the only owner of a residual it has not delivered.

Scope narrowed 2026-08-03, before implementation. As filed this bundled a config fix, a diagnostics fix, and an open-ended memory investigation — three deliverables one PR cannot cleanly close.

Out of Scope

• Changing the authority-lease contract. #16230's TTL liveness is correct and deliberately ignores pid; this ticket only improves one error message.
• The host-edge orchestrator. It holds a different per-role lease file by construction and is not implicated.
• The plane's staleness. kb-server and mc-server report efe4490dd7; bringing the plane current is #16454's lane, not this one.
• Any third-party deployment. The authority lease does not exist at the revisions any external deployment currently runs, so this specific loop cannot be occurring there.

Avoided Traps

• Filing the lease refusal as the bug. It is the loudest line in the log and it is a symptom. Two earlier readings of this incident — a pid-probe false positive, then a permanent self-deadlock — were both wrong, and the lease descriptor's own lastPulse progression is what falsified them.
• Raising the heap ceiling and closing. That converts a 3-minute crash loop into a longer one if the growth is unbounded. The ceiling is the mitigation; the growth is the question.
• Treating docker ps health as truth. It reported Up 2 minutes (healthy) during this investigation while docker exec simultaneously refused with "container is restarting". RestartCount and the exec result are the reliable reads.

Related

• #16230 — introduced the authority lease; its TTL-not-pid liveness is correct and not the cause.
• #16283 — orchestrator healthcheck treating long tasks as daemon death; adjacent symptom surface, distinct cause.
• #16454 — the unattended update caller whose dry-run surfaced this.
• #16430 — a degraded Memory Core gating orchestrator startup; different gate, same blast radius.

Live latest-open sweep: checked latest 20 open issues at 2026-08-03T18:16Z; keyword sweep on authority lease across open+closed; no equivalent found.

Origin Session ID: 9f05cd72-5457-4ec2-926c-ef1406041f19

Retrieval Hint: query_raw_memories("orchestrator container OOM heap limit 949 restarts authority lease refusal is a symptom")