LearnNewsExamplesServices
Frontmatter
id13077
titleWake daemon: a swallowed delivery throw still advances the watermark → failed wake consumed, not retried
stateClosed
labels
bugai
assigneesneo-opus-grace
createdAtJun 13, 2026, 9:51 AM
updatedAtJun 21, 2026, 2:17 AM
githubUrlhttps://github.com/neomjs/neo/issues/13077
authorneo-opus-grace
commentsCount3
parentIssue13067
subIssues[]
subIssuesCompleted0
subIssuesTotal0
blockedBy[]
blocking[]
closedAtJun 13, 2026, 5:25 PM

Wake daemon: a swallowed delivery throw still advances the watermark → failed wake consumed, not retried

neo-opus-grace
neo-opus-grace commented on Jun 13, 2026, 9:51 AM

Context

Surfaced during the cross-family review of #13073 (Codex app-server wake adapter). Directly relevant to the wake-reliability premise of #13067 / the 2026-06-13 operator escalation ("wakes report successful delivery without reliable lane conversion"). This is a pre-existing property of the wake daemon's delivery path, shared by every adapter — #13073 inherits it consistently, so it was approved with this as a follow-up.

The Problem

A wake delivery that fails (the adapter spawn throws) is logged but the wake is still consumed, not retried — the watermark advances as if it were delivered.

Traced in ai/daemons/wake/daemon.mjs:

  1. The per-subscription deliver loop: await deliverDigest(subscription, digest) (:660), then it advances the woken-watermark to the delivered events' max logId (:662-669), so those events drop below the mark and are never re-counted.
  2. deliverDigest dispatches the adapter inside a serialized deliveryPromise.then(async () => { try { … } catch (err) { writeLog('ERROR', …) } }) (:812-:1090) and return deliveryPromise (:1093).
  3. The outer try catch (:1088) swallows the error (logs, no re-throw). So the chain resolves even on a delivery failure → :660 resolves → the watermark advances.

Net: codex CLI down / app-server unreachable / tmux send-keys failure / osascript failure → [Wake Daemon] Failed to deliver via <adapter> is logged, but the wake is marked delivered and lost.

The Architectural Reality

The catch must distinguish two cases it currently conflates:

  • Intentional skip (early return): stale target presence (:824), missing appName (:853), Codex fail-closed (:874), unresolvable instance pid (:903). These resolve normally and advancing the watermark is correct — don't retry a stale/misconfigured target forever.
  • Delivery failure (a throw from the adapter spawn): a real/transient failure to a live target. Advancing the watermark here is the bug — the wake should be retried, not consumed.

The Fix (direction, not prescription)

  • Have deliverDigest report delivery outcome (delivered / intentionally-skipped / failed) rather than swallowing all non-skip errors uniformly.
  • Gate the :662 watermark advance on delivered (and skipped), not on failed.
  • Bound the retry to avoid a storm: a persistently-failing target (e.g. Codex not running) must not re-attempt every 3s POLL_INTERVAL forever. Cap retries / apply backoff, and after the cap, advance the watermark + emit a loud terminal error (so it fails visibly and doesn't wedge the loop).

Acceptance Criteria

  • A delivery failure (adapter throw) does NOT advance the woken-watermark on the first failure — the wake is retried.
  • Intentional skips (stale target / missing appName / Codex fail-closed / unresolvable pid) DO advance the watermark (unchanged).
  • Bounded retry / backoff so a persistently-failing target does not storm every poll; after the cap, the watermark advances + a terminal ERROR is logged.
  • Applies to all adapters (osascript / tmux / webhook / codex-app-server) — the gating lives in the shared path, not per-adapter.
  • Unit test: a failing adapter spawn (mocked) does NOT advance the watermark on first failure (re-queued); a capped persistent failure eventually advances + logs terminal.

Related

Surfaced in #13073 review (review PRR_kwDODSospM8AAAABC7CjOw). Reliability-aligned with #13067 (Codex app-server adapter — the adapter whose "fail visibly" intent this completes) and #11829 (anti-idle-out wake substrate). Distinct from delivery-routing; this is delivery-confirmation.

Release classification: post-release (wake-delivery reliability hardening; not v13.x-blocking).

Authored by Claude Opus 4.8 (Claude Code, @neo-opus-grace / Grace).

tobiu referenced in commit 5d5ae44 - "fix(wake): retry failed wake deliveries via a bounded retry store (#13077) (#13096) on Jun 13, 2026, 5:25 PM
tobiu closed this issue on Jun 13, 2026, 5:25 PM