LearnNewsExamplesServices
Frontmatter
id14179
titleGlobal systemic-fault circuit-breaker — recognize a cross-collection embedder outage as ONE fault, suppress the mass-heal storm
stateClosed
labels
bugaiarchitectureai-generated
assigneesneo-opus-grace
createdAtJun 27, 2026, 2:33 AM
updatedAtJun 28, 2026, 1:19 AM
githubUrlhttps://github.com/neomjs/neo/issues/14179
authorneo-opus-grace
commentsCount2
parentIssuenull
subIssues[]
subIssuesCompleted0
subIssuesTotal0
contentTrust
projected
quarantined0
signals[]
blockedBy[]
blocking[]
closedAtJun 28, 2026, 1:19 AM

Global systemic-fault circuit-breaker — recognize a cross-collection embedder outage as ONE fault, suppress the mass-heal storm

Closed v13.1.0/archive-v13-1-0-chunk-7 bugaiarchitectureai-generated
neo-opus-grace
neo-opus-grace commented on Jun 27, 2026, 2:33 AM

The gap (V-B-A'd 2026-06-26)

The actuator's safety bounds are all per-collection: healActionDispatch's cooldown/rate is keyed on (action, collection); acceptedLossSettlement's systemic-fault bound is a per-collection residue ratio. None see the global pattern. So a single embedder outage (the live KB-fire class — embedder down/404 across the corpus) that makes many collections fail at once → each classifies as independent drift → the actuator could fire N heals (a mass re-embed storm) against the already-down provider, deepening the outage and churning every collection. Grep confirms no global/cross-collection guard in healActionDispatch / recoveryRunStateStore / RecoveryActuatorService; no existing ticket.

Premise (operatorless cloud)

#14134 already states the principle per-collection: "a mismatch/loss RATE above threshold = a misconfigured embedder, not bounded drift → freeze + record, NEVER a mass auto-re-embed." This extends it to the global/cross-collection scope: N-collections-failing-at-once = ONE systemic embedder fault, not N independent drifts. Without it, the "runs for weeks unattended" bar is exposed to a self-inflicted mass-heal storm during exactly the outages the immune system exists to survive.

Fix shape

A pure global pre-dispatch decider (the cross-collection analogue of acceptedLossSettlement's per-collection bound): given the per-collection diagnosis outcomes for a sweep + a global bound (failing-collection count OR ratio of the corpus) + a window, decide:

  • independent-drifts (below the global bound) → proceed to per-collection heal/dispatch as normal;
  • global-systemic-fault (at/above the bound) → SUPPRESS the mass per-collection heal, warm-provider, freeze the affected set, and record one systemic-fault to the heal-ledger (#14163) — never mass-re-embed against a down provider.

Composes with: the per-collection dispatch bounds (#14143); #14166 freeze re-probe (the frozen set auto-recovers when the global fault clears); #14163 ledger (the single systemic-fault record).

Scope

The pure decider is buildable now — its input is the per-collection diagnosis list DataIntegrityDiagnosisService.gatherAndDiagnose() already produces (a settled shape), so this is not speculative. The integration into the diagnosis→dispatch path lands with the apply()-cutover (#14138 + the service heal-wiring). v13.1 — the fleet-scale safety guard.

Parent #14134. Refs #14039 · #14146 (the KB-fire that motivates it) · #14163 (the ledger sink) · #14166 (the freeze it triggers).

Authored by Grace (Claude Opus 4.8, Claude Code). Session 090a68e6-1a28-4b20-a5fd-842ebac3e729.

tobiu referenced in commit 42e925d - "feat(ai): cross-collection systemic-fault circuit-breaker decider (#14251) (#14252) on Jun 27, 2026, 7:18 PM
tobiu referenced in commit 342d723 - "feat(ai): persist the dispatch heal outcome to the heal-event ledger (#14260) (#14262) on Jun 27, 2026, 8:46 PM
tobiu referenced in commit eb54f64 - "feat(ai): wire the systemic-fault circuit-breaker — suppress the cross-collection mass-heal storm (#14179) (#14275) on Jun 28, 2026, 1:19 AM
tobiu closed this issue on Jun 28, 2026, 1:19 AM