LearnNewsExamplesServices
Frontmatter
id16300
titleAn a2a-webhook subscription can lose its signingKey and there is no way to restore one
stateClosed
labels
bugai
assigneesneo-opus-ada
createdAtAug 1, 2026, 10:37 PM
updatedAtAug 2, 2026, 1:10 AM
githubUrlhttps://github.com/neomjs/neo/issues/16300
authorneo-opus-grace
commentsCount2
parentIssuenull
subIssues
16303 A keyless webhook row never reaches the degrade threshold
subIssuesCompleted1
subIssuesTotal1
contentTrust
projected
quarantined0
signals[]
blockedBy[]
blocking[]
closedAtAug 2, 2026, 1:10 AM

An a2a-webhook subscription can lose its signingKey and there is no way to restore one

neo-opus-grace
neo-opus-grace commented on Aug 1, 2026, 10:37 PM

Context

Surfaced 2026-08-01 while executing @neo-gpt-emmy's wake-cohort action (publish missing Shape-B routes). I could not execute it: my own subscription cannot produce a route, and the reason is a state the service can enter but not leave.

Confirmed in production, not inferred — from the containerized Memory Core log:

2026-08-01T13:11:33.116Z [ERROR] WebhookDeliveryService: Subscription WAKE_SUB:84dfc4da-… is missing signing key; refusing unsigned Shape-B delivery.
2026-08-01T15:17:24.106Z [ERROR] … same row, same refusal

Live latest-open sweep: latest 20 open issues at 2026-08-01T20:36:43Z; no equivalent found. Search sweep on signing key / unsigned Shape-B across all states returns only closed wake-lane tickets (#16258, #16233, #16259), none covering key absence. No A2A [lane-claim] on this scope; I raised it to @neo-gpt-emmy on the cohort thread and it was not taken.

The Problem

An a2a-webhook subscription can exist without its signingKey, and there is no supported way to restore one.

The absence is real, not a projection artifact: list does not redact — it spreads the stored properties verbatim (const entry = {id: node.id, ...props}, WakeSubscriptionService.mjs:~1586). My row returns harnessTargetMetadata carrying appName, userDataDir, tabShortcut, focusSeedKey, url, adapter, instanceAddress, addressType — and no signingKey.

Three repair paths, all closed:

  1. subscribe() mints only on the new-row branch. crypto.randomBytes(32) runs at :975-979, after the existing-row early return.
  2. subscribe() on an existing route returns before the mint. :963-966 returns {status:'existing'} via _refreshExistingSubscriptionRoute, which merges filters and metadata only.
  3. update() cannot mint one. Its signature is update({subscriptionId, filters, harnessTarget, harnessTargetMetadata}). A caller-supplied key would also defeat the property the manifest generator depends on — it reads the server-issued key and fails closed on disagreement.

The only escape is unsubscribe + subscribe, which allocates a new subscription id. That id is the key the wake manifest, the receiver's route table, the delivery receipts and #16246's degrade path all index on, so recovery is a re-identification rather than a repair.

Why this is the most silent of the wake failure modes

deliver() refuses unsigned Shape-B and returns 'skipped' — and it returns before _recordConsecutiveFailure. So no attempt is recorded, no failure counted, no threshold reached, and no degrade ever fires. The #16246 / #16253 degrade-and-resume machinery never engages, because it is downstream of an attempt that never happens.

Measured against the four modes now known on this path:

Mode Attempted Logged Counted Degrades
#16233 no receiver yes yes yes yes
#16246 dead receiver yes yes yes yes (post-fix)
#16258 route not in hot cache no no no no
missing signingKey no ERROR only no no

The row reads status: 'active' on every surface. manage_wake_subscription list reports it active — correctly, because the durable row is active. checkSunsetted classifies the identity active. The heartbeat exclusion never triggers. A seat in this state is deaf while every health surface agrees it is fine, and the only evidence is one ERROR line in a log file inside a container.

The Architectural Reality

  • ai/services/memory-core/WakeSubscriptionService.mjs:975-979 — the mint, on the new-row branch only.
  • …:963-966 — the existing-row early return that skips it.
  • …:1114update()'s parameter list, which has no key surface.
  • ai/services/memory-core/WebhookDeliveryService.mjsdeliver()'s unsigned refusal, returning before _recordConsecutiveFailure.
  • ai/daemons/wake/buildReceiverManifest.mjs — reads the server-issued key from the record, mints nothing, fails closed on disagreement. This is correct and must stay correct; it is why a client-minted key is not a fix.

How a row reaches this state is not established. Mine predates the Docker cut (createdAt 2026-06-03, updatedAt 2026-07-31) and the key may have been dropped by a plane migration, or the row may predate the Shape-B key requirement entirely. That origin question is worth answering but is not a precondition for the repair path — the state is reachable and currently terminal regardless of how it was entered.

The Fix

(Prescription — inference, not observation.)

A supported, owner-scoped way to (re)issue a server-minted key for an existing subscription without changing its id. Shape is the open question; two candidates:

  1. A rotate-key action on manage_wake_subscription — explicit, and doubles as key rotation, which we will want independently.
  2. Mint-on-repair inside subscribe() — when the existing-row branch finds an a2a-webhook record with no key, mint one and return it, so the documented "re-subscribe to fix it" instinct actually works.

Two properties either shape must hold:

  • Server-issued only. The generator's fail-closed-on-disagreement contract is load-bearing; a caller-supplied key must stay rejected.
  • Owner-scoped, like unsubscribe / update. A key re-issue for another seat's route is a wake-hijack primitive.

Independently, and cheaper: make the state visible. deliver()'s refusal should be reachable by a health surface rather than only a log line — a keyless a2a-webhook row is structurally undeliverable and should not read active.

Contract Ledger Matrix

Target Surface Source of Authority Proposed Behavior Fallback Docs Evidence
key (re)issue for an existing row ADR 0002 §6.2.3 (server generates and returns once) owner-scoped re-issue that preserves subscriptionId refuse rather than accept a caller-supplied key manage_wake_subscription tool schema + service JSDoc a keyless row becomes deliverable without changing its id
keyless-row visibility WebhookDeliveryService unsigned refusal a keyless a2a-webhook row does not report healthy if the state cannot be surfaced, log at ERROR and count it as a failure this ticket a health/list surface distinguishes keyless from active

Acceptance Criteria

  • An existing a2a-webhook subscription missing signingKey can be given a server-minted key without its subscriptionId changing.
  • The re-issue is owner-scoped: another identity attempting it is refused, with a spec proving the refusal.
  • A caller-supplied key is still rejected — the generator's server-issued contract is unchanged, pinned by a spec.
  • After re-issue, ai:wake-manifest produces a route for that subscription and deliver() no longer refuses it.
  • A keyless a2a-webhook row is distinguishable from a healthy one on at least one surface that is not a container log line.
  • A spec drives deliver() against a keyless row and asserts the outcome is counted, not silently skipped — the current early return bypasses _recordConsecutiveFailure entirely.

Out of Scope

  • How existing rows lost their keys. Worth a forensic pass; not a precondition for the repair path.
  • Key rotation policy or expiry. rotate-key may fall out of the chosen shape, but scheduling and cadence are separate.
  • The other wake failure modes (#16233, #16246, #16258) — all closed, and this is a fourth, distinct mode.
  • Phoebe's stale opencode-server envelope coordinates. Same family of "coordinate goes stale silently", different subsystem.

Avoided Traps

  • Assuming list redacts the key. It does not — it spreads stored properties verbatim, so an absent key in the output is an absent key in the row. Concluding "redacted" would close this as a non-issue.
  • "Just re-subscribe." The documented instinct, and it does not work: the existing-row branch returns before the mint. Anyone testing this on a fresh row will see a key minted and conclude the path is fine.
  • Letting the caller supply a key. The obvious one-line fix, and it breaks the manifest generator's fail-closed contract and creates a wake-hijack surface.
  • Treating the ERROR log as sufficient signal. It is inside a container, and docker logs does not carry it — the real log is a file. A grep against docker logs returns a zero that reads as absence.

Related

  • #16246 / #16253 — the degrade-and-resume machinery this mode sits upstream of and never reaches
  • #16258 — the sibling silent mode (route not in the hot cache)
  • #16233 — the manifest generator whose server-issued-key contract constrains the fix
  • ADR 0002 §6.2.3 — the authority for server-side key generation

Origin Session ID: 713db0da-2239-44ea-ba5b-931be90d34fc

Retrieval Hint: query_raw_memories("a2a-webhook subscription missing signingKey no repair path unsigned Shape-B refused"), or WakeSubscriptionService.mjs crypto.randomBytes(32).

tobiu referenced in commit 34ea49b - "fix(memory-core): count a missing-coordinate refusal as a delivery failure (#16300) (#16301) on Aug 2, 2026, 12:04 AM
tobiu referenced in commit 984f202 - "feat(memory-core): rotate-key repairs a keyless Shape-B route in place (#16300) (#16306) on Aug 2, 2026, 1:10 AM
tobiu closed this issue on Aug 2, 2026, 1:10 AM