LearnNewsExamplesServices
Frontmatter
id14291
titleSupervisor defer-policy tasks respawn into a held port → EADDRINUSE
stateClosed
labels
bugaiarchitecture
assigneesneo-opus-grace
createdAtJun 28, 2026, 2:21 PM
updatedAtJun 28, 2026, 4:58 PM
githubUrlhttps://github.com/neomjs/neo/issues/14291
authorneo-opus-grace
commentsCount0
parentIssuenull
subIssues[]
subIssuesCompleted0
subIssuesTotal0
contentTrust
projected
quarantined0
signals[]
blockedBy[]
blocking[]
closedAtJun 28, 2026, 4:58 PM

Supervisor defer-policy tasks respawn into a held port → EADDRINUSE

Closed v13.1.0/archive-v13-1-0-chunk-7 bugaiarchitecture
neo-opus-grace
neo-opus-grace commented on Jun 28, 2026, 2:21 PM

Context

The local orchestrator's ProcessSupervisor tried to (re)start the Neural Link Bridge and crashed on a port collision:

[ProcessSupervisor] Starting Neural Link Bridge (supervisor-restart).
[ProcessSupervisor] Bridge: Startup Error: Error: listen EADDRINUSE: address already in use :::8081

A Bridge was already bound to :::8081, yet the supervisor spawned another. Operator-reported (2026-06-28) on the local (not cloud) orchestrator. The ask: the Bridge should resume an already-running instance the way chroma does, not re-spawn into it.

The Problem

The supervisor has two "already-running → don't re-spawn" mechanisms, and a defer-policy task benefits from neither when its PID file goes stale:

  1. PID-file adoption — adopts a live process whose pidfile matches expectedCommand.
  2. Singleton-port reconcile (reapDuplicateListeners) — for the default policy it SIGKILLs non-canonical listeners on singletonPort, freeing it; for duplicateListenerPolicy: 'defer' it is a no-op (return 0).

chroma self-heals a stale-state respawn because the default reap clears the port holder first. The Neural Link Bridge is defer, so:

  • the reconcile step is a no-op (the live holder is left untracked);
  • if the pidfile is stale/missing, PID-adoption fails (state.running = false);
  • superviseTaskgateRestartOnLivenessProbe; a down/throwing probe (orphan half-dead, the IPv4 127.0.0.1 probe missing the :::8081 IPv6 bind, or a timeout) → runTask('supervisor-restart') → spawn → new WebSocketServer(:::8081)EADDRINUSE.

The deeper defect: defer's own docstring says "matching listeners are treated as externally-owned live instances" — but the implementation ignores them entirely. It never reconciles the live holder into tracked state, so the supervisor never learns it is running.

The Architectural Reality

  • ai/daemons/orchestrator/services/ProcessSupervisorService.mjs
    • reapDuplicateListeners() (~L948): deferreturn 0 — the no-op gap.
    • runTask() (~L493/497): spawns on state.running alone — no pre-spawn port check.
    • superviseTask() (~L818/833): early-returns when state.running — so adopting before this short-circuits the respawn.
    • recoverRunningTask() (~L138) + taskStateService.adoptRunning() + watchRecoveredTask() (~L117): the existing adoption seam to reuse.
  • ai/daemons/orchestrator/taskDefinitions.mjs (~L205): neuralLinkBridge = singletonPort: 8081 + duplicateListenerPolicy: 'defer' + a 127.0.0.1 livenessProbe.
  • ai/daemons/orchestrator/Orchestrator.mjs (~L816): the poll runs the reconcile step before superviseTask every cycle — adopting in the reconcile step lands before the restart decision.

The Fix

Make the per-poll singleton-port reconcile honor the defer contract: instead of ignoring a matching listener, adopt it into tracked state.

  • Rename reapDuplicateListenersreconcileSingletonPort (its own JSDoc already reads "Enforces or observes a single live process" — the name should cover both halves). Default policy = reap duplicates (unchanged); defer policy = adopt the live expectedCommand holder.
  • defer branch: if state is not already running AND listPortListeners(singletonPort) has an expectedCommand-matching PID → adoptRunning + watchRecoveredTask + write the pidfile (self-heals the stale pidfile). Never kill (defer = never reap). A foreign command on the port → log + leave it (no spawn-into, no kill).
  • This generalizes chroma-style "already-running → no re-spawn" to the port (robust to a stale/missing pidfile) for every defer task (Bridge, dev-server).
  • Optional complementary hardening: align the Bridge liveness probe with its bind (probe ::1/127.0.0.1 consistently, or bind 127.0.0.1) so a healthy existing Bridge is also detected via liveness.

Decision Record impact

none — a behavior fix to an existing supervisor service; no ADR authority touched.

Acceptance Criteria

  • A defer singleton-port task with a live expectedCommand listener on its port AND not-running tracked state is adopted (tracked running = true, pid set, pidfile written, exit watched) instead of re-spawned.
  • The default reap policy is unchanged (non-canonical port listeners still SIGKILLed; the existing reap unit tests stay green).
  • A defer task NEVER kills the port holder (the externally-owned contract).
  • After adoption, the next poll's superviseTask does not respawn (no EADDRINUSE).
  • Unit coverage in ProcessSupervisorService.spec.mjs for the defer-adopt path: live holder adopted; foreign command not adopted; already-running is a no-op.

Out of Scope

  • The cloud orchestrator (this is the local supervisor).
  • Re-architecting the liveness/health probe framework. The IPv4/IPv6 probe-parity hardening is optional and may be a follow-up.
  • Killing externally-owned Bridges (defer must never reap).

Avoided Traps

  • A new pre-spawn guard in runTask (the first-pass idea) — rejected: runTask is the shared hot path for every supervised task; the elegant fix makes the existing defer reconcile honor its own contract, leaving runTask untouched.
  • Dropping defer so the default reap kills the holder — rejected: defer exists precisely so an externally-owned / shared Bridge is never SIGKILLed by the supervisor.

Related

  • #13289 (detect stale harness/bridge processes) — adjacent (detection), distinct from this resume fix.
  • Same port-contention class as the orphaned chroma-unit-test local flake.

Handoff Retrieval Hint: "ProcessSupervisor defer singletonPort EADDRINUSE adopt reconcile"

Authored by Grace (Claude Opus 4.8, Claude Code).

tobiu referenced in commit 232c1a1 - "fix(ai): defer-policy supervisor — adopt the live port holder + gate restart on port occupancy (#14291) (#14292) on Jun 28, 2026, 4:58 PM
tobiu closed this issue on Jun 28, 2026, 4:58 PM