Systemic-fault circuit-breaker decider — decideSystemicCircuit

Context

#14179 (the global systemic-fault circuit-breaker) needs a decision core: the cross-collection detection + circuit state-machine that recognizes a shared embedder outage as ONE fault. This leaf is slice-1 of #14179 — the pure decider — built + pushed on grace/14179-systemic-circuit-breaker (commit 161432d01). It gives slice-1's PR a clean standalone Resolves target; #14179 itself is Refs (not fully delivered until the wiring slice lands).

V-B-A (against dev): decideHealAction (healActionDispatch.mjs) filters recent runs by (run.action === action && run.collection === collection) — strictly per-(action,collection). A shared embedder outage (404 / ECONNREFUSED) fails EVERY collection's re-embed heal independently; the per-collection gate cannot see the correlation, so N collections each retry within their own bounds — a mass-heal storm hammering a dead embedder.

The Problem

The per-collection safety gate is structurally blind to a SYSTEMIC fault (one shared dependency down). Without a cross-collection layer, the autonomous actuator amplifies an embedder outage into a storm — the opposite of safety. The fix is a sibling decider that correlates failures ACROSS collections.

The Architectural Reality

• New: ai/services/memory-core/helpers/healSystemicCircuit.mjs — sibling to healActionDispatch.mjs's decideHealAction. PURE (no I/O); the caller folds the heal-event ledger (healEventLedgerStore) and passes the data.
• It is a SIBLING, not an extension: decideHealAction's per-(action,collection) contract stays untouched.

The Fix

• decideSystemicCircuit({recentFailures, circuitOpenedAt, now, bounds}) → {open, status, reason, distinctFailingCollections?}. State machine: closed → tripped (≥ systemicThreshold DISTINCT collections failing with the outage signature in windowMs) → circuit-open (suppress within openDurationMs) → half-open-probe (allow one recovery probe after cooldown). Indeterminate input (non-finite clock/bounds) → indeterminate (does NOT suppress; the per-collection gate's fail-closed owns bad-clock safety).
• isEmbedderOutageFailure(detail) — the shared-dependency outage signature matcher (EMBEDDER_OUTAGE_SIGNATURE, case-insensitive). Focused on transport/endpoint outage signals so isolated data faults don't masquerade as a shared outage.
• EMBEDDER_OUTAGE_SIGNATURE + DEFAULT_SYSTEMIC_CIRCUIT_BOUNDS exported (tunable; consumed by the wiring slice).

Acceptance Criteria

• Trips OPEN when ≥ systemicThreshold DISTINCT collections fail with the outage signature in-window.
• DISTINCT requirement: one collection failing many times is NOT systemic (the per-collection anti-thrash owns that case).
• SIGNATURE requirement: distinct collections failing with non-outage (data) details do NOT trip.
• WINDOW requirement: distinct outage failures outside windowMs do NOT trip.
• Open-state machine: within openDurationMs suppresses (circuit-open); past it goes half-open-probe (does not suppress).
• Indeterminate input defers (open: false), never spuriously suppresses.
• 15/15 unit (healSystemicCircuit.spec.mjs).

(All ACs delivered by slice-1's PR — this leaf documents the slice deliverable.)

Out of Scope (= #14179 slice-2, the wiring)

• Recording the heal OUTCOME detail to the heal-event ledger (today the orchestrator records only status: 'attempt', NOT the failure detail the circuit reads) — a prerequisite for the circuit to have failure evidence.
• The circuit-open / circuit-close ledger events + their fold (→ circuitOpenedAt).
• The cross-collection recentFailures / circuitState readers + recordCircuitEvent (orchestrator wiring, extending the existing recentRunsReader / recordRun pattern).
• The DataRecoveryActuatorService open / half-open lifecycle integration (suppress when open; record open on trip; record close / re-open on probe outcome).

Decision Record impact

aligned-with ADR-0027 (the autonomous data-recovery actuator) — this is the cross-collection safety layer above its per-collection dispatch.

Related

• Refs #14179 (parent — the wired circuit-breaker; slice-2 Resolves it)
• #14163 (the heal-event ledger the circuit reads)
• #14247 (the TextEmbeddingService 404 outage mode informing the signature)

Origin Session ID

090a68e6-1a28-4b20-a5fd-842ebac3e729

Retrieval Hint

query_summaries: "systemic-fault circuit-breaker decideSystemicCircuit cross-collection embedder outage"; commit 161432d01.