Context
The local orchestrator's ProcessSupervisor tried to (re)start the Neural Link Bridge and crashed on a port collision:
[ProcessSupervisor] Starting Neural Link Bridge (supervisor-restart).
[ProcessSupervisor] Bridge: Startup Error: Error: listen EADDRINUSE: address already in use :::8081
A Bridge was already bound to :::8081, yet the supervisor spawned another. Operator-reported (2026-06-28) on the local (not cloud) orchestrator. The ask: the Bridge should resume an already-running instance the way chroma does, not re-spawn into it.
The Problem
The supervisor has two "already-running → don't re-spawn" mechanisms, and a defer-policy task benefits from neither when its PID file goes stale:
- PID-file adoption — adopts a live process whose pidfile matches
expectedCommand.
- Singleton-port reconcile (
reapDuplicateListeners) — for the default policy it SIGKILLs non-canonical listeners on singletonPort, freeing it; for duplicateListenerPolicy: 'defer' it is a no-op (return 0).
chroma self-heals a stale-state respawn because the default reap clears the port holder first. The Neural Link Bridge is defer, so:
- the reconcile step is a no-op (the live holder is left untracked);
- if the pidfile is stale/missing, PID-adoption fails (
state.running = false);
superviseTask → gateRestartOnLivenessProbe; a down/throwing probe (orphan half-dead, the IPv4 127.0.0.1 probe missing the :::8081 IPv6 bind, or a timeout) → runTask('supervisor-restart') → spawn → new WebSocketServer(:::8081) → EADDRINUSE.
The deeper defect: defer's own docstring says "matching listeners are treated as externally-owned live instances" — but the implementation ignores them entirely. It never reconciles the live holder into tracked state, so the supervisor never learns it is running.
The Architectural Reality
ai/daemons/orchestrator/services/ProcessSupervisorService.mjs
reapDuplicateListeners() (~L948): defer → return 0 — the no-op gap.
runTask() (~L493/497): spawns on state.running alone — no pre-spawn port check.
superviseTask() (~L818/833): early-returns when state.running — so adopting before this short-circuits the respawn.
recoverRunningTask() (~L138) + taskStateService.adoptRunning() + watchRecoveredTask() (~L117): the existing adoption seam to reuse.
ai/daemons/orchestrator/taskDefinitions.mjs (~L205): neuralLinkBridge = singletonPort: 8081 + duplicateListenerPolicy: 'defer' + a 127.0.0.1 livenessProbe.
ai/daemons/orchestrator/Orchestrator.mjs (~L816): the poll runs the reconcile step before superviseTask every cycle — adopting in the reconcile step lands before the restart decision.
The Fix
Make the per-poll singleton-port reconcile honor the defer contract: instead of ignoring a matching listener, adopt it into tracked state.
- Rename
reapDuplicateListeners → reconcileSingletonPort (its own JSDoc already reads "Enforces or observes a single live process" — the name should cover both halves). Default policy = reap duplicates (unchanged); defer policy = adopt the live expectedCommand holder.
defer branch: if state is not already running AND listPortListeners(singletonPort) has an expectedCommand-matching PID → adoptRunning + watchRecoveredTask + write the pidfile (self-heals the stale pidfile). Never kill (defer = never reap). A foreign command on the port → log + leave it (no spawn-into, no kill).
- This generalizes chroma-style "already-running → no re-spawn" to the port (robust to a stale/missing pidfile) for every
defer task (Bridge, dev-server).
- Optional complementary hardening: align the Bridge liveness probe with its bind (probe
::1/127.0.0.1 consistently, or bind 127.0.0.1) so a healthy existing Bridge is also detected via liveness.
Decision Record impact
none — a behavior fix to an existing supervisor service; no ADR authority touched.
Acceptance Criteria
Out of Scope
- The cloud orchestrator (this is the local supervisor).
- Re-architecting the liveness/health probe framework. The IPv4/IPv6 probe-parity hardening is optional and may be a follow-up.
- Killing externally-owned Bridges (defer must never reap).
Avoided Traps
- A new pre-spawn guard in
runTask (the first-pass idea) — rejected: runTask is the shared hot path for every supervised task; the elegant fix makes the existing defer reconcile honor its own contract, leaving runTask untouched.
- Dropping
defer so the default reap kills the holder — rejected: defer exists precisely so an externally-owned / shared Bridge is never SIGKILLed by the supervisor.
Related
#13289 (detect stale harness/bridge processes) — adjacent (detection), distinct from this resume fix.
- Same port-contention class as the orphaned chroma-unit-test local flake.
Handoff Retrieval Hint: "ProcessSupervisor defer singletonPort EADDRINUSE adopt reconcile"
Authored by Grace (Claude Opus 4.8, Claude Code).
Context
The local orchestrator's
ProcessSupervisortried to (re)start the Neural Link Bridge and crashed on a port collision:A Bridge was already bound to
:::8081, yet the supervisor spawned another. Operator-reported (2026-06-28) on the local (not cloud) orchestrator. The ask: the Bridge should resume an already-running instance the waychromadoes, not re-spawn into it.The Problem
The supervisor has two "already-running → don't re-spawn" mechanisms, and a
defer-policy task benefits from neither when its PID file goes stale:expectedCommand.reapDuplicateListeners) — for the default policy it SIGKILLs non-canonical listeners onsingletonPort, freeing it; forduplicateListenerPolicy: 'defer'it is a no-op (return 0).chromaself-heals a stale-state respawn because the default reap clears the port holder first. The Neural Link Bridge isdefer, so:state.running = false);superviseTask→gateRestartOnLivenessProbe; adown/throwing probe (orphan half-dead, the IPv4127.0.0.1probe missing the:::8081IPv6 bind, or a timeout) →runTask('supervisor-restart')→ spawn →new WebSocketServer(:::8081)→ EADDRINUSE.The deeper defect:
defer's own docstring says "matching listeners are treated as externally-owned live instances" — but the implementation ignores them entirely. It never reconciles the live holder into tracked state, so the supervisor never learns it is running.The Architectural Reality
ai/daemons/orchestrator/services/ProcessSupervisorService.mjsreapDuplicateListeners()(~L948):defer→return 0— the no-op gap.runTask()(~L493/497): spawns onstate.runningalone — no pre-spawn port check.superviseTask()(~L818/833): early-returns whenstate.running— so adopting before this short-circuits the respawn.recoverRunningTask()(~L138) +taskStateService.adoptRunning()+watchRecoveredTask()(~L117): the existing adoption seam to reuse.ai/daemons/orchestrator/taskDefinitions.mjs(~L205):neuralLinkBridge=singletonPort: 8081+duplicateListenerPolicy: 'defer'+ a127.0.0.1livenessProbe.ai/daemons/orchestrator/Orchestrator.mjs(~L816): the poll runs the reconcile step beforesuperviseTaskevery cycle — adopting in the reconcile step lands before the restart decision.The Fix
Make the per-poll singleton-port reconcile honor the
defercontract: instead of ignoring a matching listener, adopt it into tracked state.reapDuplicateListeners→reconcileSingletonPort(its own JSDoc already reads "Enforces or observes a single live process" — the name should cover both halves). Default policy = reap duplicates (unchanged);deferpolicy = adopt the liveexpectedCommandholder.deferbranch: ifstateis not already running ANDlistPortListeners(singletonPort)has anexpectedCommand-matching PID →adoptRunning+watchRecoveredTask+ write the pidfile (self-heals the stale pidfile). Never kill (defer = never reap). A foreign command on the port → log + leave it (no spawn-into, no kill).defertask (Bridge, dev-server).::1/127.0.0.1consistently, or bind127.0.0.1) so a healthy existing Bridge is also detected via liveness.Decision Record impact
none— a behavior fix to an existing supervisor service; no ADR authority touched.Acceptance Criteria
defersingleton-port task with a liveexpectedCommandlistener on its port AND not-running tracked state is adopted (trackedrunning = true, pid set, pidfile written, exit watched) instead of re-spawned.defertask NEVER kills the port holder (the externally-owned contract).superviseTaskdoes not respawn (no EADDRINUSE).ProcessSupervisorService.spec.mjsfor the defer-adopt path: live holder adopted; foreign command not adopted; already-running is a no-op.Out of Scope
Avoided Traps
runTask(the first-pass idea) — rejected:runTaskis the shared hot path for every supervised task; the elegant fix makes the existingdeferreconcile honor its own contract, leavingrunTaskuntouched.deferso the default reap kills the holder — rejected:deferexists precisely so an externally-owned / shared Bridge is never SIGKILLed by the supervisor.Related
#13289(detect stale harness/bridge processes) — adjacent (detection), distinct from this resume fix.Handoff Retrieval Hint: "ProcessSupervisor defer singletonPort EADDRINUSE adopt reconcile"
Authored by Grace (Claude Opus 4.8, Claude Code).