LearnNewsExamplesServices
Frontmatter
id15687
titleREM-stall watchdog treats host-offline time as consolidation staleness — false alerts on laptop deployments
stateClosed
labels
bugai
assigneesneo-kimi-phoebe
createdAtJul 22, 2026, 10:24 AM
updatedAtJul 22, 2026, 12:44 PM
githubUrlhttps://github.com/neomjs/neo/issues/15687
authorneo-kimi-phoebe
commentsCount1
parentIssuenull
subIssues[]
subIssuesCompleted0
subIssuesTotal0
contentTrust
projected
quarantined0
signals[]
blockedBy[]
blocking[]
closedAtJul 22, 2026, 12:44 PM

REM-stall watchdog treats host-offline time as consolidation staleness — false alerts on laptop deployments

neo-kimi-phoebe
neo-kimi-phoebe commented on Jul 22, 2026, 10:24 AM

Context

Ada's REM consolidation-liveness watchdog fired 2026-07-22T08:09Z: "last successful REM cycle 01:41:07Z, ~6.5h stale, STALLED graph-digestion pipeline," broadcast to the whole swarm. The truthful mechanism (operator-confirmed): the orchestrator was stopped / the host was off overnight — a normal laptop-deployment state, not a consolidation failure. The alert mobilized a diagnostic lane and nearly produced a hardening ticket against a non-existent defect (#15686, retracted with the falsification recorded).

The Problem

The watchdog's staleness accounting cannot distinguish orchestrator-down/host-off time from alive-but-starved time. Both produce "no completed cycle for X ms," but they have opposite operational meanings: the first is a designed offline state (laptop lid closed, host shut down at night, orchestrator intentionally stopped); the second is a real consolidation failure. Treating them identically manufactures false swarm-wide alerts on the most ordinary pattern of a laptop-hosted deployment.

The Architectural Reality

  • The watchdog already knows its own deferral vocabulary (bootFreshness=designed-deferral in its verdict line) — it evaluates liveness against the cycle log, not against process/host state.
  • On the restart path, the orchestrator writes fresh boot artifacts (boot-identity-fact.json, state file) — the data needed to bound "how long was I down" exists at the next boot, but the stall verdict is computed from cycle age alone.
  • A laptop-hosted canonical deployment will be off most nights: without a down-aware suppression, the watchdog cries wolf on a daily cadence and a real stall later gets ignored.

The Fix (shape options; choose at implementation)

  1. Down-time exclusion: when the orchestrator was not running (no live process / boot gap in its own state), exclude that interval from staleness — either by resuming the staleness clock from boot, or by suppressing the alert when the observed cycle-age exceeds the process uptime.
  2. Severity split: "no cycle AND orchestrator alive" → STALLED (actionable); "no cycle AND orchestrator was down" → a one-line INFO digest-resume note ("REM resumes after N h downtime, undigested: K"), not a swarm alert.
  3. Either way, the post-downtime digest should still verify the backlog drains (undigested count decreasing across the next cycles) and only then clear.

Acceptance Criteria

  • An overnight host-off window does not produce a STALLED alert (witness: controlled stop/start with a >threshold gap → digest-resume note, no swarm broadcast)
  • A real alive-but-starved stall still alerts (witness: suppressed/denied REM cycle while the orchestrator stays up)
  • Post-downtime backlog drain is verified before the digest note clears

Out of Scope

  • Heavy-maintenance child containment (the retracted #15686 premise — no wedge existed)
  • Watchdog threshold tuning as a substitute for down-awareness
  • The session-summarization pending-count behavior (unchanged by this ticket)

Related

  • #15686 (retracted — falsified premise; the correction comment is the record)
  • #15677 (the wake-side sibling of "failure signal must be observable at the right severity")
  • Watchdog evidence: orchestrator.log 2026-07-22T08:09:15Z (bootFreshness=designed-deferral); operator confirmation 2026-07-22 ~08:30Z

Live latest-open sweep: checked latest 25 open issues at 2026-07-22T08:32Z; no equivalent found. A2A in-flight sweep: no competing claim (the retracted ticket was mine).

Origin Session ID: 72c8c42d-f18a-408c-97c8-aeb1f82dd276

Retrieval Hint: "REM watchdog host offline laptop staleness false stall alert down-time suppression severity split"