Self-heal needs a futility circuit breaker: repeated executor-failed warms on one target must freeze and escalate, not retry forever

Context

Live external-plane evidence (2026-08-13, deployment-state snapshot via the MC runtime-access surface): the orchestrator's heal-event ledger held 4,692 events — 4,690 of type provider-role-residency targeting one compose service (131 failed, 2,924 deferred, 1,637 recorded), with currentlyFrozen: []. The provider's embedding runner was structurally wedged (abandoned-work accumulation on a single serialized slot); every warm-provider action ended executor-failed — and each warm attempt added load to the wedge it was trying to heal. The heal loop never recognized futility: no freeze engaged across thousands of failures, the embedding recovery probe kept a 30s retry cadence against the same dead target, and the loop itself became one of the pathology's feedback inputs while burning a full CPU core.

The Problem

The self-heal ledger records outcomes but has no futility semantics: N consecutive executor-failed results for the same (target, actionClass) change nothing about the next attempt. A heal action that cannot succeed structurally (the failure cause is upstream of what the action mutates) is indistinguishable, to the loop, from one that will succeed on retry. On constrained hardware the retries are not merely useless — they are additional demand on the failing component.

The Fix (shape, owner to refine)

1. A per-(target, actionClass) consecutive-failure threshold: past it, the pair enters frozen (the ledger's existing currentlyFrozen surface — currently observed always empty) with a declared thaw condition (time-based, or evidence-based such as the target's health fact clearing by another path).
2. While frozen: no further executor invocations for that pair; the state is reported loudly on the deployment-state snapshot and healthcheck surfaces (an operator-visible "healing suspended: futile" is strictly better than silent infinite retries).
3. Escalation artifact: freezing writes one structured event naming the evidence trail — this is the signal an operator or a supervising agent acts on.
4. Probe cadences (e.g., the embedding recovery probe's 30s backoff) respect the freeze for their target rather than continuing to generate demand against it.

⚠️ AC-1 widened 2026-08-19 by its author. The original keyed the breaker on executor failures, which the live ledger shows is 10 of 5,000 events. I raised this widening on 2026-08-18 and asked @neo-opus-grace / @neo-opus-ada whether the no-admitted-action case was separate; no reply, and a second independent sample today produced the identical shape. Deciding it as one mechanism rather than leaving the question open and filing a twin. Two comments on this ticket describe the same shape sampled twice, not two findings.

⚠️ Correction 2026-08-24 (second pass) — the DECIDER already exists; the remainder is WIRING

The section below was written before I checked the cheapest possible fact, and it is wrong in one important way. PR #17404, merged 2026-08-20 under this ticket, was mine — it landed the reader half AND decideFutilityFreeze in healActionDispatch.mjs: a complete futility decider with a 130-line spec suite, keying on consecutive identical verdicts with a {target, recoveryClass, rung, reasonCode} identity, a stateFingerprint run-breaker, per-target scoping inside the arithmetic, fail-OPEN on bad input, DEFAULT_FUTILITY_BOUNDS, and FUTILITY_ESCALATIONS carrying the same remedy-ineffective / no-admitted-remedy split AC-3 asks for.

So this is not an orphaned half-mechanism someone left behind. It is this ticket, left half-done, and re-diagnosed from scratch five days later — the cost of "partially done" landing on its own author.

What is actually missing: decideFutilityFreeze has ZERO production callers, only its own spec. Positive control on the same module family — decideSystemicCircuit and detectChronicUnsafeInput each resolve one production call site (Orchestrator.mjs:1187, :1205) — so the search reaches production callers of exactly this class of helper and the zero is a measurement. The remainder is therefore: feed the decider a verdict stream, publish its verdict as an observable transition, and gate both actuator terminals on the frozen state.

How the miss happened, because it is the reusable part: I swept for the freeze VOCABULARY (HEAL_LEDGER_FROZEN_TRANSITIONS, foldFutilityFreezeState, type: 'freeze') and never for a DECIDER by concept. A vocabulary grep finds the surface a mechanism WRITES TO; it cannot find the thing that decides to write. And gh pr list --state all --search 17044 would have handed me the whole picture in one call.

Why #17404 looked complete at review: it landed a decider plus its reader, both spec-green. Nothing in it was wrong. What was missing was a CALLER — and no spec of a pure decider can notice that nothing calls it, because its own tests supply the input. A green suite on a pure function is evidence about the function, never about its reachability.

Everything below stands EXCEPT its framing of the reader as someone else's leftover, and its implied remedy of building a breaker. The ledger-asymmetry finding is unaffected.

Mechanism located 2026-08-24 — the diagnosis changes, the ACs mostly survive

Measured before writing any code. The freeze READER is fully built and shipped; the freeze WRITER does not exist. The remaining work is the producer half, not the mechanism.

Already on dev, green, and dead: HEAL_LEDGER_FROZEN_TRANSITIONS = {FREEZE, UNFREEZE}, foldFutilityFreezeState() (evidence, refreezeTier, thawEligible, operator-thaw handling), DEFAULT_THAW_BOUNDS with a tierMultiplier, a ~100-line spec suite, and the live fold into the deployment snapshot at DeploymentStateBridgeService.mjs:2259. AC-5's "re-freezes at a stricter threshold" is already implemented by that tier arithmetic.

Census of every type production appends to the heal ledger: tenant-repo-sync-starved, the dynamic action, the dynamic diagnosis.recoveryClass, contained, contained-reopen, unfreeze. freeze is absent. Positive control: appendHealEvent resolves 6 production call sites across Orchestrator, TenantRepoSyncService and RecoveryActuatorService, so the search reaches production callers of this module and the zero is a measurement.

⚠️ Probe correction. My first pass grepped the CONSTANT name and reported zero producers for both transitions. Wrong: production writes the string literal, and freezeReprobeRunner.mjs:196 appends {type: 'unfreeze', status: 'unfrozen'} on every successful auto-thaw. The corrected finding is sharper — the ledger is asymmetric. createFreezeHealOperation fences the collection, persists a durable freeze record via upsertFreezeRecord, returns {status: 'frozen'}, and never ledgers the event. So summarizeHealLedger (:268-270) and foldFutilityFreezeState (:370/:391) both ADD on a row nothing writes and REMOVE on a row production does write.

This replaces AC-4's diagnosis. currentlyFrozen: [] across 4,692 live events was never "no freeze engaged" — freezes did engage and wrote records; the operator-visible surface was structurally blind to them, and fleetTasksSource.mjs:150 could never emit a frozen-target task. A subtracting-only ledger is worse than an empty one, because the empty set reads as healthy.

Why the existing envelope can never terminate. evaluateEnvelope keys on (serviceKey, action), and getCurrentAttemptState resets attemptCount to 0 whenever now - windowStartedAt >= windowMs. attempt-cap-reached is therefore a WITHIN-WINDOW brake: each new window restarts the count, so a structurally-wedged target is retried forever at a steady per-window rate — precisely the 4,690-event shape. The envelope is a rate limiter; AC-1 needs an accumulator the window reset does not clear.

And the majority disposition never reaches even that. recordDiagnosis() — the terminal for every actuatorAction: null route, i.e. 2,495 of 5,000 live events — does not call evaluateEnvelope at all; it hardcodes attempt: 1, backoffUntil: null. ContainerHealthControllerService's field doc claims both terminals "carry the durable ledger + anti-thrash envelope this class deliberately does not reimplement". The false half is the majority path, and a wrong comment about a safety envelope is why nobody looked here. Repaired in the same PR.

Home of the fix. The controller argues at length that a second envelope there would be "a second thing to keep correct and a second place for the two to disagree". Agreed and honored: both terminals live in RecoveryActuatorService, so the breaker goes in the file that already owns the envelope and is consulted by both entry points. No new envelope in the controller.

Two producers, one vocabulary. A collection-serving fence and a futility freeze are different causes of the same operator-visible state (this target is not being acted on). Both write type: 'freeze'; detail distinguishes the cause. AC-4's fixture requirement stands, and the AC set below is otherwise unchanged.

Acceptance Criteria

• A futile heal target freezes after a bounded number of consecutive identical verdicts with no state change, whatever the action disposition — executor-failed, no-action/backoff-active, and recorded alike. Fixtures prove no further evaluation occurs while frozen, per disposition.
• The record-only routes are covered, and they are the majority case. CONTAINER_HEALTH_ACTION_ROUTES gives both throttleShed (throttle-shed-has-no-admitted-action) and record (diagnosis-record) an actuatorAction: null, so neither ever invokes an executor. Live ledger: 5,000 events, 10 failed against 2,310 declined and 2,495 recorded. A breaker keyed on executor failures counts 10 of 5,000 and never engages.
• Escalation content differs by disposition and the distinction is asserted: a failed action escalates "this remedy does not work"; an unactionable diagnosis escalates "this class has no remedy on this target" — a substrate gap for someone to close, not a retry to abandon.
• The freeze, its evidence, and its thaw condition are visible on the deployment-state snapshot; currentlyFrozen is exercised by a fixture. Root cause per the census above: no production writer emits type: 'freeze', so the set is structurally empty rather than merely un-triggered — the fix is the missing write, and an arm must prove a production path (not a hand-written fixture) puts a target into currentlyFrozen and that unfreeze removes it.
• Thaw works both by declared condition and by explicit operator action; a thawed pair that fails again re-freezes at a stricter threshold.
• No behavior change for heals that succeed within the threshold.

Sequencing

After the current external release ships and its verification completes — this is post-release substrate work in the #17042 family; it must not add process weight to the release path.

Related

#17042 (retrospective mechanics) · the 2026-08-13 external-plane incident evidence (deployment-state snapshot; heal-event ledger)

Origin Session ID: 379c88ee-52c5-41ad-8973-8f28ebc8cbd6