LearnNewsExamplesServices
Frontmatter
id11766
titleFold the swarm-heartbeat into the Orchestrator as a local-only lane
stateClosed
labels
enhancementairegressionarchitecture
assigneesneo-opus-ada
createdAtMay 22, 2026, 3:46 PM
updatedAtMay 22, 2026, 7:28 PM
githubUrlhttps://github.com/neomjs/neo/issues/11766
authorneo-opus-ada
commentsCount0
parentIssue11730
subIssues[]
subIssuesCompleted0
subIssuesTotal0
blockedBy[]
blocking[]
closedAtMay 22, 2026, 7:28 PM

Fold the swarm-heartbeat into the Orchestrator as a local-only lane

Closed v13.0.0/archive-v13-0-0-chunk-13 enhancementairegressionarchitecture
neo-opus-ada
neo-opus-ada commented on May 22, 2026, 3:46 PM

Context

The swarm-heartbeat daemon is currently down. Diagnosed 2026-05-22: no installed launchd plist in ~/Library/LaunchAgents/; no running heartbeat process; the heartbeat-*.log files in .neo-ai-data/wake-daemon/ are stale since ~2026-05-04; the last line of heartbeat.log is swarm-heartbeat.sh: line 181: claude: command not found.

The daemon is broken at two independent layers: (1) the launchd job was never installed — the .plist is a template requiring manual [OPERATOR_SUBSTITUTE_REPO_ROOT] substitution; (2) the swarm-heartbeat.sh wrapper assumes claude is on PATH, which launchd does not provide. A down heartbeat caused a swarm idle-out on the night of 2026-05-21 (zero PRs that window) — with no operator present and the heartbeat down, agents that go idle are not woken back. The interim wake mechanism is GPT's Codex-harness cron plus operator presence.

The Problem

The launchd .plist + swarm-heartbeat.sh approach is tech debt, independent of the current outage:

  • The .plist is a manual-substitution template — fragile install, easy to skip, no drift detection.
  • The .sh wrapper carries PATH assumptions that fail under launchd.
  • It is a second local daemon lifecycle to manage, parallel to the Orchestrator.

Patching the plist/wrapper would invest in debt that should be retired. The Agent OS already has a local supervisor — the Orchestrator — and the heartbeat is intrinsically a local-supervision concern (it wakes idle local maintainer-swarm agents). The heartbeat should be an Orchestrator lane, not an orphan daemon.

The Architectural Reality

  • The Orchestrator (ai/daemons/Orchestrator.mjs) is the local Agent OS supervisor — a mixed-responsibility daemon with cloud-relevant and local-only lanes, per Epic #11720 / ADR 0014 (learn/agentos/decisions/0014-*), whose scheduler-task-taxonomy classifies daemon lanes cloud-deployable / local-only.
  • The swarm-heartbeat is SwarmHeartbeatService (ai/daemons/SwarmHeartbeatService.mjs) + the swarm-heartbeat-daemon.mjs entry-point wrapper + the swarm-heartbeat.sh bash wrapper + the com.neomjs.swarm-heartbeat.plist launchd template (learn/agentos/wake-substrate/).
  • #11761 (merged 2026-05-22) converted SwarmHeartbeatService to a dual-mode export — SwarmHeartbeatService.pulse() is now cleanly importable, which makes folding it into the Orchestrator a direct import, not a subprocess.
  • #11758 established the NEO_ORCHESTRATOR_*_ENABLED config pattern — the cloud profile pins orchestrator local-only lanes off. The heartbeat lane follows the same pattern.

The Fix

Fold the heartbeat into the Orchestrator and retire the standalone-daemon machinery:

  1. FoldSwarmHeartbeatService.pulse() becomes an Orchestrator-scheduled lane (the Orchestrator imports and drives it on its existing scheduler), eliminating the separate swarm-heartbeat-daemon.mjs process.
  2. Config-gate — the heartbeat lane is gated by a config flag (NEO_ORCHESTRATOR_HEARTBEAT_ENABLED-class, following the #11758 pattern), default-off in the cloud profile, default-on in the local profile. The flag MUST be a genuine config default — never a hardcoded / build-time exclusion or tight local-only-path coupling (see Avoided Traps + the forward-compat note in Out of Scope).
  3. Retire — remove the com.neomjs.swarm-heartbeat.plist launchd template and the swarm-heartbeat.sh bash wrapper. Consolidate to at most one minimal launchd entry, for the Orchestrator itself.
  4. ADR — record the heartbeat lane's classification (local-only-default, config-seam) in an ADR 0014 amendment or a follow-up ADR, per the #11720 ADR owner-map pattern.

Contract Ledger

Target Surface Source of Authority Proposed Behavior Fallback / Edge Case Docs Evidence
Heartbeat-lane enable config flag (Orchestrator config; NEO_ORCHESTRATOR_HEARTBEAT_ENABLED-class) This ticket; #11758 NEO_ORCHESTRATOR_*_ENABLED precedent; ADR 0014 lane taxonomy Boolean. Default-ON in the local profile, default-OFF in the cloud profile. A genuine toggle — flipping it on post-v13 needs no redesign. If unset, resolve to the local-profile default (heartbeat on). Orchestrator config docs + ADR 0014 amendment Unit coverage for per-profile default resolution
com.neomjs.swarm-heartbeat.plist template + swarm-heartbeat.sh Retired by this ticket Removed — the Orchestrator owns the heartbeat lane n/a — superseded Retirement noted in the wake-substrate docs Files deleted; grep-clean of references

Acceptance Criteria

  • SwarmHeartbeatService.pulse() runs as an Orchestrator-scheduled lane; the standalone swarm-heartbeat-daemon.mjs process is no longer required.
  • The heartbeat lane is gated by a config flag — default-off in the cloud profile, default-on in the local profile — implemented as a config default, NOT a hardcoded exclusion.
  • The com.neomjs.swarm-heartbeat.plist launchd template and swarm-heartbeat.sh wrapper are removed; no dangling references remain (grep-clean).
  • Under the local profile, a running Orchestrator drives heartbeat pulses (verifiable via heartbeat-log / pulse evidence).
  • Under the cloud profile, the heartbeat lane does not run (config-disabled).
  • The heartbeat lane's local-only-default classification + config seam is recorded in an ADR 0014 amendment or follow-up ADR.
  • Unit coverage for the config-flag per-profile default resolution.

Out of Scope

  • The in-deployment agent runtime. Post-v13, agents could run inside a container deployment, which would flip the heartbeat lane back on. This ticket does NOT design or build that runtime — the config flag is the forward-compat seam that keeps it open at zero cost. Building the in-deployment runtime now would threaten the v13 release.
  • Wake-subscription duplicate-accumulation (#11182) — adjacent wake-substrate concern, separately tracked.
  • Re-homing other orchestrator daemons / non-heartbeat lanes.

Avoided Traps

Trap Why rejected
Patch the launchd .plist / fix the .sh wrapper Invests in tech debt being retired; leaves the second-daemon-lifecycle problem intact
Hardcode "heartbeat is local-only" (build-time exclusion / tight local-path coupling) Calcifies a current-scope default into an architectural law — post-v13 agents-in-container re-enable becomes a rewrite. The cloud-disable must be a config default
Keep the heartbeat a separate process, Orchestrator merely supervising its lifecycle Retains a process to manage; the fold (heartbeat = an Orchestrator lane) eliminates the separate daemon entirely. #11761's dual-mode export makes the fold a clean import
Per-daemon launchd plists Scatters local-daemon lifecycle; consolidate to one Orchestrator entry

Related

  • Parent: #11730 (Cloud Agent OS Deployment — Post-MVP Residual Workstreams)
  • #11720 — Cloud Agent OS Deployment Readiness (the MVP epic, completed); ADR 0014 (learn/agentos/decisions/0014-*)
  • #11761 — wake-script dual-mode export (merged 2026-05-22) — makes SwarmHeartbeatService.pulse() cleanly importable
  • #11758NEO_ORCHESTRATOR_*_ENABLED cloud-profile config pattern
  • #11182 — Wake subscription duplicate-accumulation (adjacent wake-substrate bug, distinct)

Origin Session ID

ff79d594-1c1e-4181-ad9b-3d9150547699

Handoff Retrieval Hints

  • query_raw_memories(query="swarm heartbeat fold into Orchestrator local-only lane")
  • query_raw_memories(query="swarm-heartbeat daemon down launchd plist tech debt 2026-05-22")
  • Memory anchor: the 2026-05-22 operator design conversation that converged the Orchestrator-owned heartbeat direction (heartbeat diagnosed down; fold-not-supervise; config-default-not-hardcode forward-compat seam).
tobiu referenced in commit 84c128e - "feat(ai): fold swarm-heartbeat into the Orchestrator as a local-only lane (#11766) (#11772) on May 22, 2026, 7:28 PM
tobiu closed this issue on May 22, 2026, 7:28 PM
tobiu referenced in commit 86bd0c6 - "feat(agentos): bump swarmHeartbeatMs default 5m → 15m (#11840) (#11841) on May 23, 2026, 2:39 PM