LearnNewsExamplesServices
Frontmatter
titledocs(agentos): add wake substrate incident protocol (#10650)
authorneo-opus-ada
stateMerged
createdAtMay 3, 2026, 6:24 PM
updatedAtMay 3, 2026, 7:00 PM
closedAtMay 3, 2026, 7:00 PM
mergedAtMay 3, 2026, 7:00 PM
branchesdevagent/10650-wake-restart-incident-protocol
urlhttps://github.com/neomjs/neo/pull/10655
Merged
neo-opus-ada
neo-opus-ada commented on May 3, 2026, 6:24 PM

Authored by Claude Opus 4.7 (1M context) (Claude Code). Session 9766f91c-51f8-44fe-ac34-d79f61a0e1bf.

Resolves #10650 Related: #10647 (parent epic), #10601 (grandparent epic), #10648 (gate primitive — merged), #10649 (matrix sibling — Gemini owns)

The operator-facing companion to the wake safety gate (#10648 / merged in PR #10653) and the cross-harness prompt-landing matrix (#10649 / Gemini's lane). Codifies the discipline today's 2026-05-03 incident exposed as missing: incident declaration, freeze rule, restart preflight checklist (including @neo-gpt's bridge backlog fence), reactivation evidence requirements, coordination mode, ownership split, and a post-incident retrospective template.

Surface

  • learn/agentos/WakeSubstrateIncidentProtocol.md (new, 211 lines) — protocol document itself.
  • learn/tree.json — indexed under the AgentOS top-level, between agentos/CodeExecution and AgentOS/Measurements. Validated as valid JSON via jq.

What the protocol covers

Per the ticket's "The Fix" section + GPT's bridge-backlog-fence addition (MESSAGE:b90e0c31-cdca-4b1e-9da0-2eb69a65df34, captured as a hard preflight requirement in tobi's green-light):

  1. When to Declare an Incident — five trigger conditions matched against today's failure modes (orphan-spawn, payload outside prompt surface, cross-harness divergence, operator manual intervention, repeated regression).
  2. The Freeze Rule — no individual local fix authorizes wake-substrate reactivation; references the 8-layer wake substrate loop diagnosis from #10647.
  3. Restart Preflight Checklist — process inventory (PIDs), gate state, subscription inventory, and the bridge backlog fence: lastSyncId vs max GraphLog.log_id vs pending count. If pending > 0, operator MUST choose ONE of three documented paths (disable subscriptions / advance lastSyncId / targeted test bridge) before bridge starts. Empirical anchor: 893 rows pending in this incident.
  4. Reactivation Evidence Requirements — gate enabled + #10649 matrix green + local regressions merged + backlog fence path chosen, OR explicit operator override.
  5. Coordination Mode During an Incident — durable mailbox (add_message / list_messages) is the path-asymmetric substrate that survives wake-delivery offline state.
  6. Ownership Split — local bug owners fix tickets; one reactivation-gate owner coordinates loop-level validation.
  7. Post-Incident Retrospective Template — symptom / failed layer / proof / fix tickets / validation evidence / recurrence guard (the load-bearing field — without a recurrence guard, the retrospective is incomplete).
  8. Empirical Anchor: 2026-05-03 Incident — the canonical retrospective populated using the template, citing all the failure layers (Layer 6 Antigravity Cmd+L toggle, Layer 3 Codex validAppNames + AgentIdentity cache strip, Layer 7 staleness-as-sunset) with the manual-kill PID list + fix tickets + validation status.

Deltas from ticket

None substantive. Bridge backlog fence is in the protocol checklist as a hard preflight (per @neo-gpt's bridge-backlog-fence A2A + tobi's green-light explicitly named it as a hard preflight requirement). All ticket AC items map cleanly:

  • AC: protocol exists + linked from #10647 → Related Tickets section in the doc
  • AC: restart is a release event → "When to Declare an Incident" + "Restart Preflight Checklist"
  • AC: defines when to disable/keep disabled heartbeat + wake → "The Freeze Rule"
  • AC: live inventory of PIDs / MCP state / subscriptions / harness targets → "Restart Preflight Checklist"
  • AC: requires #10649 matrix evidence → "Reactivation Evidence Requirements"
  • AC: requires #10648 gate to be in place → "Reactivation Evidence Requirements"
  • AC: names durable A2A mailbox as fallback → "Coordination Mode During an Incident"
  • AC: incident-retrospective template → "Post-Incident Retrospective Template"

Test Evidence

Documentation-only PR; no runtime code paths. Verification:

$ jq '.' learn/tree.json > /dev/null && echo "valid JSON"
valid JSON

$ wc -l learn/agentos/WakeSubstrateIncidentProtocol.md
211 learn/agentos/WakeSubstrateIncidentProtocol.md

Per pr-review-guide §7.5: "If the PR is a documentation or template change, no tests are required." Manual link-validation review: all #NNNNN references use the canonical issue/PR URL form; all agentos/... cross-doc references resolve to existing files in learn/agentos/.

Post-Merge Validation

  • Knowledge Base re-indexes the new doc on next sync (npm run ai:sync-kb or equivalent)
  • ask_knowledge_base(query: "wake substrate incident protocol") returns the new doc
  • Future incident retrospectives cite this protocol's template structure (organic adoption test)

Out of Scope (per ticket-stated scope)

  • #10648 circuit breaker implementation — already merged in PR #10653
  • #10649 matrix runner implementation — Gemini's lane
  • Fixing local regressions (#10643, #10645, #10633, #10627) — separate substrate work
  • Reactivating heartbeat — explicitly NOT this PR; reactivation requires the criteria documented in this protocol's "Reactivation Evidence Requirements" section

Related Coordination Anchors

  • @tobiu green-light A2A relayed via @neo-gpt MESSAGE:82bfdf32-5ac0-4305-b21d-b789095fa591
  • Bridge backlog fence A2A from @neo-gpt MESSAGE:b90e0c31-cdca-4b1e-9da0-2eb69a65df34
  • Initial split proposal A2A from @neo-gpt MESSAGE:b4aed6a9-9c6a-48df-841b-a2dd12359871
  • My epic-review on #10647 establishing acceptance: https://github.com/neomjs/neo/issues/10647#issuecomment-4366482408