LearnNewsExamplesServices
Frontmatter
id16366
titleA reload-lag 404 permanently degrades the wake route it is arming
stateClosed
labels
bugaiarchitecture
assigneesneo-opus-grace
createdAtAug 2, 2026, 4:38 PM
updatedAtAug 2, 2026, 5:28 PM
githubUrlhttps://github.com/neomjs/neo/issues/16366
authorneo-opus-grace
commentsCount0
parentIssuenull
subIssues[]
subIssuesCompleted0
subIssuesTotal0
contentTrust
projected
quarantined0
signals[]
blockedBy[]
blocking[]
closedAtAug 2, 2026, 5:28 PM

A reload-lag 404 permanently degrades the wake route it is arming

Closed Backlog/active-chunk-12 bugaiarchitecture
neo-opus-grace
neo-opus-grace commented on Aug 2, 2026, 4:38 PM

Context

Found by @neo-opus-vega while restoring her own wake route (broadcast 13:51Z), and routed to me to judge because the degrade mechanism is mine (#16246, closed). Her incident is the empirical anchor; this ticket is the guard she asked for. Live latest-open sweep 2026-08-02T14:40Z — nearest neighbours are #16352 and #16267, both distinct (see Relationship below).

The Problem

WebhookDeliveryService.mjs:146-151 degrades a route on any 4xx, with no retry and no threshold:

if (response.status >= 400 && response.status < 500) {
    // 4xx: client error, no retry, mark degraded immediately
    await this._markDegraded(subscription.id);
    return 'skipped';
}

For 400/401/403 that is right — those are unambiguous statements that the route is invalid.

404 unknown-subscription is not unambiguous. The receiver emits it for two incompatible reasons, and by design:

receiver's meaning source correct verdict
this route was deliberately withdrawn buildReceiverManifest.mjs:316-321 — an empty manifest is "the correct end state for a fully-unsubscribed seat: the receiver answers 404 unknown-subscription" degrade
I have not reloaded the manifest that contains it yet receiver.mjs:242 — the route table is read per request, but only from the manifest this process has loaded do not degrade

The receiver cannot distinguish them — it only knows "not in my current manifest." The sender cannot either, from the status code alone. Today the sender resolves the ambiguity toward degrade, on first sight, permanently.

Why this is worse than a delayed wake

Degradation is terminal by design (WebhookDeliveryService.mjs:235) — a degraded route is never probed again, which is exactly what bounds the attempt count and is the property #16246 shipped for. Applied to a reload-lag 404, that same property converts a timing gap into a permanent outage that outlives its own cause.

@neo-opus-vega measured the full sequence on this plane:

13:12Z  route e9e8e1e2 published to the manifest
        receiver still serving the manifest it loaded at 12:53Z → 404 unknown-subscription → degraded
13:14Z  operator restarts the receiver WITH the route present
        route still dead: resume returned wasDegraded: true

Nineteen minutes of gap produced a dead route that fixing the receiver did not revive. The more carefully an operator publishes before signalling, the wider the window they open. That is a self-defeating workflow, and it is a plausible (unconfirmed) explanation for other seats reading active while silent.

The Fix

Route 404 unknown-subscription through the existing consecutive-failure threshold instead of degrading on first sight. Every other 4xx stays immediately terminal.

This is the shape the file already chose for its other ambiguous case, and the reasoning transfers verbatim (WebhookDeliveryService.mjs:98-101):

"Counted through the same threshold rather than degraded on first sight, deliberately: the existing machinery already owns persistence, restart survival, and the resume path, and a second degrade trigger with its own semantics would be a second thing to keep correct."

Consequences, both directions:

  • Withdrawn route — still degrades, after 3 messages instead of 1. #16246's bound survives: bounded attempts, then terminal.
  • Reload lag — a delivery that succeeds resets the counter, so the route survives its own publication window.

Detection reads the receiver's own discriminator ({error: 'unknown-subscription'} vs {error: 'not-found'}). An unparseable body falls back to today's immediate degrade — the new tolerance must be earned by a recognised signal, never granted by a parse failure.

Acceptance Criteria

  • A 404 carrying error: 'unknown-subscription' does not degrade on first sight; it counts through _recordConsecutiveFailure.
  • It still degrades once the threshold is met — a genuinely withdrawn route remains terminal and bounded.
  • A successful delivery between two such 404s resets the counter, so a route that outlives the window is never degraded for it.
  • 400/401/403, and 404 not-found (wrong path), degrade immediately as today. A spec asserts the boundary, not just the new branch.
  • A 404 whose body is absent, non-JSON, or carries an unrecognised error degrades immediately — fail-closed to current behaviour.
  • Specs are red before the change: the first assertion must fail against dev, or it is not testing the defect.

Relationship to neighbouring tickets

  • #16352 (receiver follows the manifest) closes the window; this closes the consequence. Independent and complementary: any reload mechanism has non-zero latency, so the guard still earns its place after #16352 lands. Neither blocks the other.
  • #16267 (route published after boot is permanently deaf, CLOSED) fixed the receiver's half — the route table is now per-request and reloadable. The sender's half is untouched, which is why the incident above still happened after it merged.
  • #16303 (a keyless webhook row never reaches the degrade threshold) is the mirror defect: that one never degrades when it should; this one degrades when it should not.

Out of Scope

  • Exact disambiguation via a receiver-reported manifest generation. It would make the verdict precise rather than heuristic, but it needs a receiver protocol change and #16352 may make the window small enough not to warrant it. Worth revisiting only if the threshold proves insufficient.
  • Any change to what degraded means, or to the resume path (#16253/#16255 own that).

Avoided Traps

  • Exempting 404 entirely. That reintroduces #16246 — a withdrawn route would retry forever. The threshold is what keeps it terminal.
  • Trusting the status code alone. 404 not-found and 404 unknown-subscription are different failures behind one number; only the body separates them.
tobiu referenced in commit e3fdafc - "fix(memory-core): a reload-lag 404 counts toward the threshold instead of degrading the route (#16366) (#16367) on Aug 2, 2026, 5:28 PM
tobiu closed this issue on Aug 2, 2026, 5:28 PM