LearnNewsExamplesServices
Frontmatter
id16303
titleA keyless webhook row never reaches the degrade threshold
stateClosed
labels
bugai
assigneesneo-opus-ada
createdAtAug 1, 2026, 11:32 PM
updatedAtAug 2, 2026, 12:04 AM
githubUrlhttps://github.com/neomjs/neo/issues/16303
authorneo-opus-ada
commentsCount0
parentIssue16300
subIssues[]
subIssuesCompleted0
subIssuesTotal0
contentTrust
projected
quarantined0
signals[]
blockedBy[]
blocking[]
closedAtAug 2, 2026, 12:04 AM

A keyless webhook row never reaches the degrade threshold

neo-opus-ada
neo-opus-ada commented on Aug 1, 2026, 11:32 PM

Context

Split out of #16300 (@neo-opus-grace) so each half has a close-target it genuinely resolves. #16300 owns the repair path (rotate-key); this owns detectability, which is independent of it and needs no design decision.

Filed because the PR body lint is right on principle: a PR carrying only Refs has no close-target, and the honest fix is a ticket the work actually closes rather than a Resolves pointed at something the PR does not finish.

Live latest-open sweep at 2026-08-01T21:31:30Z, latest 10: no equivalent — #16302 is off-host backups, different subsystem. A2A claim sweep over the last 6: no overlapping claim.

The Problem

WebhookDeliveryService.deliver() refuses a row missing signingKey (or url) and returns before recording anything:

if (!signingKey) {
    logger.error(`… refusing unsigned Shape-B delivery.`);
    return 'skipped';                                    // ← :97
}
…
await this._recordConsecutiveFailure(subscription.id);   // ← :155, never reached

No attempt recorded ⇒ no failure counted ⇒ no threshold met ⇒ no degrade.

The #16246/#16253 degrade machinery is not broken. It is downstream of an attempt that does not happen. That distinction is the whole defect: instrumenting the degrade path finds it healthy and concludes the subsystem is fine.

Consequence: the row reads active on every surface — list, checkSunsetted, heartbeat inclusion — while the seat receives nothing. Verified in production by @neo-opus-grace on her own a2a-webhook row.

The Architectural Reality

  • ai/services/memory-core/WebhookDeliveryService.mjs:91-98 — the two early returns (url, signingKey), both 'skipped', both unrecorded.
  • :155_recordConsecutiveFailure, reached only after the retry loop exhausts.
  • :166-174 — the threshold (3) and _markDegraded, which own persistence and restart survival.
  • ai/services/memory-core/CoalescingEngineService.mjs:401 — the only consumer that branches on the outcome string, and only on === 'delivered'.

The Fix

Record a consecutive failure at both refusals and return 'failed' rather than 'skipped'.

Two properties to keep:

  1. Still issue no request. The refusal must stay a refusal; making it louder must not make it attempt an unsigned delivery.
  2. Reuse the existing threshold rather than adding a second degrade trigger. That machinery already owns persistence, restart survival, and #16253's resume path.

Acceptance Criteria

  • A row missing signingKey degrades after the existing threshold instead of never.
  • A row missing url behaves identically — the sibling branch is not left behind.
  • Neither refusal issues an HTTP request (the pre-existing invariant survives).
  • The degrade write is observable (properties.status === 'degraded'), so the row stops reading active.
  • Reverting the recording makes the specs fail — the falsifier is behavioural, not absent-capability.

Out of Scope

  • The repair path. rotate-key stays on #16300; this ticket makes the state visible, it does not fix it.
  • Degrading on first sight. A missing key is permanent rather than transient, so immediate degrade is arguable — deliberately not taken, because a second trigger with its own semantics is a second thing to keep correct. Cost is up to two more silent wakes: bounded and self-clearing, unlike forever.
  • How rows lose their keys. Forensics, tracked with #16300.

Avoided Traps

  • Treating this as a degrade-machinery bug. It is not; the degrade is correct and unreached. Fixing it there would be a change with no effect.
  • Fixing signingKey and leaving url. Identical shape three lines up. Fixing one and leaving its twin is how the next incident gets filed.
  • Changing the outcome string without checking consumers. 'skipped''failed' is only safe because a sweep showed one consumer branching on 'delivered' alone.

Related

  • #16300 — the repair half (rotate-key), same subsystem
  • #16246, #16253 — the degrade and resume machinery this makes reachable

Origin Session ID: 56105163-6e66-44b6-8c6f-9e81bc1be08c

Retrieval Hint: query_raw_memories("keyless webhook row skipped never reaches degrade threshold"), or ai/services/memory-core/WebhookDeliveryService.mjs:95.

tobiu closed this issue on Aug 2, 2026, 12:04 AM