LearnNewsExamplesServices
Frontmatter
id16882
titleEvery daemon reports a crash as success: uncaughtException exits 0, so a container cannot tell a crash from a graceful stop
stateClosed
labels
bugai
assigneesneo-opus-grace
createdAtAug 10, 2026, 2:21 PM
updatedAtAug 10, 2026, 4:12 PM
githubUrlhttps://github.com/neomjs/neo/issues/16882
authorneo-opus-grace
commentsCount0
parentIssuenull
subIssues[]
subIssuesCompleted0
subIssuesTotal0
contentTrust
projected
quarantined0
signals[]
blockedBy[]
blocking[]
closedAtAug 10, 2026, 4:12 PM

Every daemon reports a crash as success: uncaughtException exits 0, so a container cannot tell a crash from a graceful stop

neo-opus-grace
neo-opus-grace commented on Aug 10, 2026, 2:21 PM

Context

Found 2026-08-10 while diagnosing a CPU-only cloud deployment whose orchestrator showed restartCount: 12 with exitCode: 0, oomKilled: false, error: null on every exit. Those readings are what a graceful shutdown looks like, so the restarts read as "somebody stopped it twelve times" rather than "it crashed twelve times". They are indistinguishable at the container level, and the reason is in our source.

Plan-Authority: INDEPENDENT — runtime defect, no governing epic. Relevant to #16706's external-plane diagnosability but not scoped by it.

The Problem

All four long-lived daemons route their uncaughtException handler into the same cleanup() that SIGTERM uses, and that cleanup() calls bare process.exit(). Node's process.exit() with no argument exits 0. Nothing in these paths sets process.exitCode.

daemon handler cleanup() exit
ai/daemons/orchestrator/daemon.mjs :203 :197 process.exit()0
ai/daemons/embed/daemon.mjs :279 :273 process.exit()0
ai/daemons/message/daemon.mjs :212 :206 process.exit()0
ai/daemons/wake/daemon.mjs :518 :512 process.exit()0

Two of them carry the comment // Calls process.exit() automatically, which documents that it exits and not with what.

The consequence is that a crash is reported as success to everything downstream of the process boundary:

  • A container runtime cannot distinguish a crash from an operator stop. restart: unless-stopped restarts either way, so the daemon does come back — the loss is not availability, it is attribution.
  • Any alerting, healing, or supervision keyed on a non-zero exit never fires. On the deployment that surfaced this, the self-heal ledger reads total: 0 with no events ever recorded, beside a container that had restarted twelve times.
  • docker inspect gives the operator exitCode: 0 and error: null, so the natural next step — "nothing crashed, check who restarted it" — is the wrong investigation, pointed away from the stack trace that was written seconds earlier.

Availability is not the defect; observability is. The restart policy masks it, which is exactly why it has survived.

The Architectural Reality

  • The three non-orchestrator daemons share a near-identical cleanup() (idempotent via cleanedUp, unlink-own-pidfile, exit). The orchestrator's differs slightly (Orchestrator.stop(), removePidFile()) but ends the same way.
  • SIGINT/SIGTERMcleanup() is correct at exit 0. A signal-initiated stop is success. Only the uncaughtException arm is wrong, so the repair must not flatten the two into one code.
  • process.on('exit', removePidFile) in the orchestrator means pid-file cleanup already survives an exit-code change.
  • Non-zero exits do exist elsewhere in these files (embed:52/254/300, message:33/189/227, orchestrator:455, hostEdge:74), so the codebase's convention is that a failure exits non-zero — these four crash paths are the deviation, not the norm.

The Fix

  1. The uncaughtException arm exits non-zero. Pass an explicit failure code through cleanup(code) (defaulting to 0) so the signal path keeps exit 0 and the crash path does not. The same for any unhandledRejection handler that shares the path.
  2. Do it in all four, and prove the shared shape rather than fixing one and copying by eye. Three of the four cleanup() bodies are already textually identical; a repair that lands in one and not the others reproduces this ticket in a year.
  3. Keep the stack log. It is already written before cleanup() and is the only surviving evidence today; nothing about the exit-code change should reorder it.

Contract Ledger

Target Surface Source of Authority Proposed Behavior Fallback Docs Evidence
daemon crash exit this ticket uncaughtException exits non-zero unchanged pid-file cleanup and stack log JSDoc on each cleanup() stating which code each caller yields unit: a handler-invoked cleanup yields non-zero
daemon signal exit existing behavior SIGINT/SIGTERM continue to exit 0 unchanged same JSDoc unit: signal path still yields 0 — the non-vacuity arm

Acceptance Criteria

  • Each of the four daemons exits non-zero when its uncaughtException handler runs, proven per-daemon. A spec that covers one daemon and asserts the others by inspection does not satisfy this.
  • The signal path still exits 0 — asserted in the same spec. Without this arm the change could set every exit non-zero and still go green, which would make every graceful stop look like a crash and is strictly worse than today.
  • The pid file is still removed on the crash path, and the stack trace is still written before exit.
  • The two misleading // Calls process.exit() automatically comments state the resulting code.
  • No new leaf is introduced. If a configurable code is proposed instead of a constant, an ADR-0019 §3 self-audit is recorded in the PR — but a constant is preferred: an exit code is a contract with the runtime, not a deployment preference.

Out of Scope

  • Why any particular daemon threw. This ticket makes crashes visible; it diagnoses none of them.
  • Restart policy, backoff, or supervision changes.
tobiu referenced in commit ee76787 - "fix(ai): a daemon crash exits non-zero so a supervisor can see it (#16882) (#16887) on Aug 10, 2026, 4:12 PM
tobiu closed this issue on Aug 10, 2026, 4:12 PM