Prove the orchestrator heap ceilings hold, and whether ~500MB is a leak

Context

Successor to #16459, created at @neo-gpt-emmy's review of PR #16460 to hold its L4 residual. That PR delivers what is provable on a branch — per-process heap ceilings, the container budget, and the lease-refusal diagnostics — and its evidence tops out at L2 (unit + config render). The two remaining claims cannot be proven by any amount of CI, because they require a running orchestrator that does not currently exist in a healthy state.

Splitting them out rather than deferring them inside #16459 is the point: a PR must not auto-close the only owner of a residual it has not delivered.

The Problem

Two things are asserted by the fix and demonstrated by nothing yet.

1. That the ceilings actually stop the loop. #16459 was observed at RestartCount=968 over ~30 hours. The fix is derived from the failure signature — death at ~500 MB, two concurrent Node processes, the child larger than the parent — but no run has yet survived under it.
2. That ~500 MB is a bounded working set rather than a leak. If it is unbounded growth, the fix converts a ~3-minute crash loop into a slower one and the real defect is untouched. #16459 says so explicitly; nothing has tested it.

The reproducer is unusually cheap while it lasts: a process that dies every ~3 minutes gives two clean crash-to-crash cycles in under ten minutes. That reproducer disappears the moment the fix works — which is the argument for capturing the profile early rather than after the plane is healthy.

The Architectural Reality

• ai/deploy/docker-compose.yml — the orchestrator's command:-scoped parent ceiling and the 3g container budget (parent 1024 + up to TWO supervised children at 384, the orchestrator.supervisedTaskHeapMb leaf default). Corrected 2026-08-04: this line said "one supervised child 512" from filing until now, which is not what a4ad9aca71 shipped — an L4 run against that number would have measured a configuration that does not exist.
• ai/daemons/orchestrator/services/ProcessSupervisorService.mjs — buildSupervisedTaskEnv(), the explicit child ceiling and its precedence chain.
• ai/daemons/orchestrator/services/MaintenanceBackpressureService.mjs:18-30 — the 11 heavy tasks the lease serialises; the assumption "at most one heavy child at a time" that the budget rests on.
• #16462 — why nobody noticed for 30 hours. Whether this verification is itself observable depends on that ticket, not this one.

The Fix

Run it and measure it.

1. Recreate the orchestrator against the merged ceilings and confirm RestartCount stops climbing while it carries its normal workload — including at least one heavy-maintenance task, since that is when the second Node process exists.
2. Capture heap profiles across two cycles (or across a sustained run once stable) and compare retained sets. State plainly which it is: bounded working set, or growth.
3. If growth: file the leak with the retained-set evidence. That is a different defect from the one the ceilings address, and it deserves its own ticket rather than being folded back here.
4. Check the budget assumption empirically — that concurrent Node processes stay at parent + one supervised child under real load. The heavy-maintenance lease is the reason to expect it; a measurement is the reason to believe it.

Contract Ledger Matrix

Decision Record impact

none — verification of an already-merged change. It changes no contract; it establishes whether one holds.

Acceptance Criteria

• The orchestrator is recreated against the merged ceilings and RestartCount is static across a workload window that includes at least one heavy-maintenance task.
• Heap profiles are captured across two cycles (or a sustained stable run) and compared.
• The ticket records the verdict explicitly: bounded working set or growth. "It stopped crashing" is not the verdict — absence of a crash within a window is not evidence of boundedness.
• If growth: a leak ticket is filed carrying the retained-set evidence.
• The concurrent-Node-process count is measured under load, counting only process-producing paths that survive the scheduling gates — the registry's supervised-child-process lane, after the one-winner-per-poll dispatch, after authority-profile and per-task enablement. A task-key count is NOT this number; an earlier revision of #16459 conflated the two and the claim was withdrawn rather than corrected.
• The measured maximum either confirms the provisional parent-1024 + two-children-at-384 budget or restates it. This deferral is owned here explicitly, not implied by a Compose comment.
• If the ceilings did not stop the loop, that is recorded as a falsification of the fix rather than retried at a larger number.

Out of Scope

• Changing the ceilings. Adjusting them is downstream of the measurement, and adjusting them instead of measuring is the trap.
• The observability gap. #16462 owns why the loop went unnoticed.
• The lease diagnostics. Delivered and unit-covered in PR #16460.

Avoided Traps

• Treating "it stopped crashing" as the answer. A slower leak also stops crashing for a while, and a window that is too short cannot tell them apart. The profile is the evidence; the absence of a crash is the prompt to take it.
• Measuring at idle. The second Node process only exists while a supervised task runs, so an idle census confirms a budget nothing is stressing.
• Letting the reproducer expire unused. A ~3-minute crash cycle is the cheapest profiling opportunity this defect will ever offer, and a successful fix destroys it.

Related

• #16459 — the ceilings and diagnostics; PR #16460.
• #16462 — the reason a 968-restart loop ran for 30 hours unnoticed.
• #16230 — the authority lease, correct and unchanged throughout.

Live latest-open sweep: checked latest open issues at 2026-08-03T19:20Z; filed as the named successor in PR #16460's review, no equivalent exists.

Origin Session ID: 9f05cd72-5457-4ec2-926c-ef1406041f19

Retrieval Hint: query_raw_memories("orchestrator heap ceiling verification bounded working set or leak two crash cycles L4 residual")